{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. CNN for Sentence Classification (Kim 2014)\n",
    "- This is Keras implementation for **CNN for Sentence Classification** paper (Kim, Y. (2014). Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882.) \n",
    "- There are four model variations\n",
    "    - ```CNN-rand```: \"Our baseline model where all words are randomly initialized and then modified during training.\"\n",
    "    - ```CNN-static```: \"A model with pre-trained vectors from word2vec. All words-including the unknown ones that are randomly initialized—are kept static and only the other parameters of the model are learned.\"\n",
    "    - ```CNN-non-static```: \"Same as above but the pretrained vectors are fine-tuned for each task.\"\n",
    "    - ```CNN-multichannel```: \"A model with two sets of word vectors. Each set of vectors is treated as a ‘channel’ and each filter is applied to both channels, but gradients are backpropagated only through one of the channels. Hence the model is able to fine-tune one set of vectors while keeping the other static. Both channels are initialized with word2vec.\"\n",
    "    \n",
    "</br>\n",
    "<img src=\"http://d3kbpzbmcynnmx.cloudfront.net/wp-content/uploads/2015/11/Screen-Shot-2015-11-06-at-8.03.47-AM.png\" style=\"width: 800px\"/>\n",
    "    \n",
    "- Dataset used is **Sentiment Analysis on Movie Reviews** from Kaggle\n",
    "    - source: https://www.kaggle.com/c/sentiment-analysis-on-movie-reviews/data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from multiprocessing import Pool\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from gensim.models import Word2Vec\n",
    "from keras.layers import *\n",
    "from keras.models import *\n",
    "from keras.utils import to_categorical\n",
    "from keras.callbacks import *\n",
    "from keras import optimizers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset import and preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(8529, 2)\n",
      "                                              Phrase  Sentiment\n",
      "0  A series of escapades demonstrating the adage ...          1\n",
      "1  This quiet , introspective and entertaining in...          4\n",
      "2  Even fans of Ismail Merchant 's work , I suspe...          1\n",
      "3  A positively thrilling combination of ethnogra...          3\n",
      "4  Aggressive self-glorification and a manipulati...          1\n"
     ]
    }
   ],
   "source": [
    "# exclude partial phrases, and import only sentences\n",
    "df = pd.read_csv(\"data\\train.tsv\", delimiter = '\\t').groupby(['SentenceId']).head(1)[['Phrase', 'Sentiment']]\n",
    "df = df.reset_index(drop = True)\n",
    "print(df.shape)\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_tokens = 10000      # number of tokens\n",
    "embed_dim = 50          # set word2vec embedding dim = 50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tokenizer = Tokenizer(num_words = num_tokens)\n",
    "tokenizer.fit_on_texts(df['Phrase'])\n",
    "sentences = tokenizer.texts_to_sequences(df['Phrase'])\n",
    "word_idx = tokenizer.word_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i in range(len(sentences)):\n",
    "    for j in range(len(sentences[i])):\n",
    "        sentences[i][j] = str(sentences[i][j])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# generate word2vec model with encoded sentences\n",
    "model = Word2Vec(sentences = sentences, size = embed_dim, sg = 1, window = 3, min_count = 1, iter = 10, workers = Pool()._processes)  \n",
    "model.init_sims(replace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sentences = pad_sequences(sentences, maxlen = len(max(sentences, key = len)), padding = 'post')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15213\n",
      "(15214, 50)\n"
     ]
    }
   ],
   "source": [
    "embedding_matrix = np.zeros((len(word_idx) + 1, embed_dim))\n",
    "\n",
    "for word, idx in word_idx.items():\n",
    "    try:\n",
    "        embedding_matrix[idx] = model[str(idx)]\n",
    "    except:\n",
    "        pass\n",
    "\n",
    "print(len(word_idx))\n",
    "print(embedding_matrix.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(6396, 47) (2133, 47) (6396, 5) (2133, 5)\n"
     ]
    }
   ],
   "source": [
    "# split data into train-test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(sentences, df['Sentiment'])\n",
    "y_train = to_categorical(y_train)\n",
    "y_test = to_categorical(y_test)\n",
    "\n",
    "print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### CNN-rand\n",
    "- All words are randomly initialized and the modified during training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def cnn_rand(embed_dim, num_tokens, max_len, windows = [3, 4, 5], filters = 100, p = 0.5, l2 = 3):\n",
    "    inputs = Input(shape = (max_len, ))\n",
    "    embedded = Embedding(input_dim = len(word_idx) + 1, output_dim = embed_dim, \\\n",
    "                         input_length = max_len, trainable = True)(inputs)\n",
    "    conv_results = []\n",
    "    for window in windows:\n",
    "        x = Conv1D(filters, window, activation = 'relu')(embedded)\n",
    "        x = Dropout(p)(x)\n",
    "        x = MaxPooling1D(max_len - window + 1)(x)\n",
    "        conv_results.append(x)\n",
    "    conv_result = concatenate(conv_results)\n",
    "    x = GlobalMaxPooling1D()(conv_result)\n",
    "    x = Dense(50)(x)\n",
    "    outputs = Dense(5, activation = 'softmax')(x)\n",
    "    m = Model(inputs = inputs, outputs = outputs)\n",
    "    m.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['acc'])\n",
    "    return m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cnn_rand = cnn_rand(embed_dim, num_tokens, X_train.shape[1])\n",
    "callbacks = [ModelCheckpoint(filepath = 'best_model.hdf5', monitor='val_acc', verbose=1, save_best_only = True, mode='max')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "history = cnn_rand.fit(X_train, y_train, epochs = 100, callbacks = callbacks, validation_split = 0.1, batch_size = 50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XeclNW9+PHPd2Ybu7Ds0qQqqCgdgRVJwIixBOxdLImY\nKNGrV425uUHv78Z4Y5oxXk2CGjVYroUQK0YUG8auFBEpIggCCwJL27477fv748zuzvZhmWXYZ77v\n14sXO8+cmTlnyvf5Puec5zyiqhhjjPEWX7IrYIwxJvEsuBtjjAdZcDfGGA+y4G6MMR5kwd0YYzzI\ngrsxxniQBXdjjPEgC+7GGONBFtyNMcaD0pL1wj169NCBAwcm6+WNMaZDWrJkyU5V7dlauaQF94ED\nB7J48eJkvbwxxnRIIrIxnnLWLWOMMR5kwd0YYzzIgrsxxnhQ0vrcmxIMBiksLKSqqirZVfGMrKws\n+vfvT3p6erKrYow5gFoN7iIyGzgD2KGqI5q4X4B7gdOACmC6qi5tS2UKCwvp0qULAwcOxD2t2R+q\nyq5duygsLGTQoEHJro4x5gCKp1vmUWBKC/dPBQZH/80A7m9rZaqqqujevbsF9gQREbp3725HQsak\noFaDu6q+A+xuocjZwOPqfATkiUiftlbIAnti2ftpTGpKRJ97P2BzzO3C6LZvEvDcSaOqRBQiqoQj\n7l8ookRUEaAmZIajZVTBJ+DzCT6BmqsXqkIEd3/DSxr6RPD73L80n490v/s7kQFZVQlFIry9Zgcl\nVSFKKoOUV4fq7gfXtrASVsUvQpq/pk51/9NUnaLvUSiihCMR9vWKjSLg9/nwC/j9vtrX84vUvpwq\nte99bf18gs8ntZ9NKBLzwjHbwhGtfY/T/ILG1FWov73mM47ENCI7M40jeuZwZM/O9OySmbDPJRJR\ndlcE2FZcRXl1iOpQhKpgmOpQJPovTG2TWniPa96/NJ97v0Lh2LZH3Pc1oq6Mv/7n6fMJgdrXi+AT\nSPf76n3ufp8Qqfe9r3vPJOYDiqd+sc8LUB2KEAhFCIYjTb5HPhE6Z6WRm5VOfnY6Rx7SmV5dshLy\n/jdUFQwTjijZGf7adqkqZdUhKgJh8rLTyUzz15ZXVapDESoCYcqjZcoDISqqw1QEQgTD7jcXjmjM\n71vqvU+De3VhRL+u7dKeGgd0QFVEZuC6bjj00EMP5Eu3KhJRCrfv5Iknn+S8y39IINT0l64p1/3g\nQn7754fJ7dr8hzXrrt8w7rhvM+H4yS0+l0+EzDQfndL9ZGX4yc9Ox+/bt0lN4UiEvRVBSqpCVAbC\nbCuu5up5i/bpOUx9g3t15ukZE+jROTPux6gqK7aUsGDlNjbtrmBbcRXflFSyvbiaQDNBzTSvR+dM\nRvTL5boTj+TYgd3iftzOsmqeXVLI3MWb+aa4iuyMNHIy/YQjyp7yAOWBMAAZaT6652SQ5heKSqup\nCtZ9RjkZfjpnpVERCFMRcDuD/XHNCUe0e3CXeC6QLSIDgX82M6D6V+BtVX06ensNMFlVW8zcCwoK\ntOEZqqtXr2bo0KFxVz4RguEIu8qq2VUeYNPGjdxw5TRef28RWRl+/OKCrUbCZGaku4xHBMVlvEBt\nGRFcph9xmUxNYiMCPtz9DTO/ekcF4QiBsBIMuyyuKhghFImQl53Bod2y42pLIBRmR2k1eyuCRFTJ\nTPOTk+Hnm01fQV4/8rPTyc1KJzszDV9MVdKi2ZUvJlOryTzCESUYbv474o/J8nz7mNhGYjLmUDhC\nOOb9iJXmr8tOIxFq6+YTId3vw+cDoe7F03yC3++OABQIh5VgJFLvSImY7RJ9D/z++m3YWxFkfVE5\nX2wr4Q8L1jCqf1eevGoCGWkt72wDoQh/e28Dzy4tZN2OMtJ8Qt+8TvTpmkXvrln06er+PiQ3i9ys\nNDLTfWT4/WSl+8hM85OR5qutY+17LNKofpEGRxzpPvdepEUzdb+4zzQc/Txrjs7C0f8z09zrpccc\n1cR+7qGI1j5Hw/csNmw09x1o6vMNhd1RbE1b3VFA4/cwHHGZc0lliF1l1XyxrZRV35Tw/rqdbCup\n4kcTB/Ef3zuarHR/4wfjdqxLN+1h9ntfs2DlNkIR5diB+Yzun0dFMExFdQgRoVtOBt1yMvD7hD3l\nAXaXBwiGI/TskknPLplkZ6SxtyLA7nJ3xNspw09Opt/tIDL8ZGemkZ3hJyczjZwM93fNEZA/9ugy\nXD+Lz8tOJy87o8XvUXNEZImqFrRWLhGZ+zzgehGZAxwHFLcW2A8Gqso3xVXsKg+AKrmd0nn47jso\n3LiBc06eSHp6OllZWeTn5/PFF1/w5Zdfcs4557B582aqqqq48cYbmTFjBlC3lEJZWRlTp05l0qRJ\nfPDBB/Tr148XX3yRzE6dmD59OmeccQYXXHABAwcO5IorruCll14iGAzyj3/8gyFDhlBUVMQVl17K\n1q1bmTBhAgtee52n579N/7xB+FqJnNXBMOt3lhOOKF07pdO9cwad0t1hZun2NIbGmen4fEKGT8iw\nUyDIzkijb14nJg3uQa/cLG54+lP++4UV/O78kS120dzx8ioe/3Ajxw7M5zfnjuT0kX3omp28qagu\nqPjJbOHXLsJB97l3yUqnT1eALnz7yB4AlFeH+O0rq3n4vQ28tWYHZ43uy8DuOQzolk1Eld3lAbaX\nVPHs0i18tnkvXTulM/3bA5k2fgBH9uqS1PYcaPFMhXwamAz0EJFC4DYgHUBVHwDm46ZBrsNNhbwy\nERW7/aWVrNpakoinqjWsby63nTkcgO2l1ewsqyY/O4NeuZlkpvn54x/uZM3qVSxbtoy3336b008/\nnRUrVtROI5w9ezbdunWjsrKSY489lvPPP5/u3bvXe421a9fy9NNP89BDD3HRRRfx7LPPcvnllzeq\nS48ePVi6dCn33Xcfd911Fw8//DC333473/3ud7nlllt49dVXmT17NpFo319up+aDQyDkArsqHNmr\nc7PZjGm7s0b35cttpfxl4TqO7t2FH05qemrpC59u4fEPNzLjO4dz62kH9ig0FeRkpnHHOSOZMrwP\nt7+0knvfXNvkWM/hPXL41dnDOX9cf7IzDqrTeQ6YVlutqpe0cr8C1yWsRgfAnooAO0qq6JadQb/8\nTs1mYePHj683P/xPf/oTzz//PACbN29m7dq1jYL7oEGDOOaYYwAYN24cX3/9dZPPfd5559WWee65\n5wB47733ap9/ypQp5Ofn4/cJxZXBZoN7MBRh/c5yIqoc3iPHAns7uvmUo1izvZRfz1/NlBG96ZvX\nqd79X2wrYeZzyxk/qBv/+b2jk1TL1DBpcA9ev/kEqoJhCvdUsGl3Bel+H/nZrpuld25Wq0e7XnfQ\n7tJqMuxEK68OUbinks6ZafRtIbAD5OTk1P799ttv88Ybb/Dhhx+SnZ3N5MmTm5w/nplZN+Dm9/up\nrKxs8rlryvn9fkKhUJNlALpkplFS5frQfU3UtXBvJeGwMqhnDp1SNEM5UHw+4RdnDOON1duZs2gz\nN59yVO19pVVBrn1iKV2y0vnLpWNI8x883RtelpXu58heXVKuyyUeKfUNjKiycXcFGX4fh3bLbhQs\nu3TpQmlpaZOPLS4uJj8/n+zsbL744gs++uijhNdv4sSJzJ07F4DXXnuNPXv2kJuVTjii9aYv1giE\nIpRWBekRHfgx7W9At2xOOKoncz7ZVG8a36yFX/H1rnJmXTq23absGbMvUiq4l1eHCIUj9Oma1WRm\n1b17dyZOnMiIESP42c9+Vu++KVOmEAqFGDp0KDNnzmTChAkJr99tt93Ga6+9xogRI/jHP/5B7969\n6d0zH78IxRXBRuX3VAQAyE/iYF0quuy4w9hRWs2bq7cDsK24ikfe38A5x/Rj/KD4p+gZ057imgrZ\nHpIxFXLz7gpKqoIM7ZPbZBdHslVXV+P3+0lLS+PDDz/k2muvZdmyZWzaVUFZtat37EkWa7aXkuH3\ncXjPzi0+bzKmmHpZKBzh+DsXcmSvzvzfj47jlueW88ySQt766WQGxDlt1Zi2OpBTITuEiColVUFy\ns9IPysAOsGnTJi666CIikQgZGRk89NBDAHTNTmNvZYDy6hCds1yWXl4dJhCKcEiudQEcaGl+H5eM\nP5S7X/+SN1dvZ+7iQr4/4TAL7OagkjLBvawqVDsH/GA1ePBgPv3000bbu2S6HdLOskD0BCRhT0UA\nv0/omnXwtsfLLj52APe+uZZ/e3IpWWk+rv/ukcmukjH1pEyfe3FlEL/PrVfR0fh8wiG5WZRUBdm4\nq4JgOEJxZZCundJTfrpXshySm8Wpww6hOhThquMP36dlCYw5EDpepGuDiCol0bniB2uXTGt6dsnE\n74MteypZuz1MRJVuOW07fdkkxnUnHokqXP2dw5NdFWMaSYngXlYVIqxKXgefVdItJxO/z8em3RVk\npfvpZCcsJdWIfl154Pvjkl0NY5qUEsG9pksmp6XFNTqIrp3SGdyrM74mFiIzxpganu9z12iXTNd2\nmCXTubObgrh161YuuOCCJstMnjyZhlM+G7rnnnuoqKiovX3aaaexd+/eZstnpfvJSLOs3RjTPM8H\n92B0qdHsdsza+/btyzPPPNPmxzcM7vPnzycvLy8RVTPGpCjPB/dAyJ2kleFvPWufOXMms2bNqr39\ny1/+kjvuuIOTTjqJsWPHMnLkSF588cVGj/v6668ZMcItdV9ZWcm0adMYOnQo5557br21Za699loK\nCgoYPnw4t912G+AWI9u6dSsnnngiJ554IuCWEN65cycAd999NyNGjGDEiBHcc889ta83dOhQrr76\naoYPH86pp57a7Bo2xpjUdPB2Qr8yE7Z9vt9PkxmJcHgwQnaGH/qMgqm/a7bsxRdfzE033cR117lF\nLufOncuCBQu44YYbyM3NZefOnUyYMIGzzjqr2f7u+++/n+zsbFavXs3y5csZO3Zs7X2//vWv6dat\nG+FwmJNOOonly5dzww03cPfdd7Nw4UJ69OhR77mWLFnCI488wscff4yqctxxx3HCCSeQn58f99LC\nxpjU5PnMvWZ1hXi628eMGcOOHTvYunUrn332Gfn5+fTu3Ztbb72VUaNGcfLJJ7Nlyxa2b9/e7HO8\n8847tUF21KhRjBo1qva+uXPnMnbsWMaMGcPKlStZtWpVi/V57733OPfcc8nJyaFz586cd955vPvu\nu0D8SwsbY1LTwZu5t5Bh74vtuysoqw4xtE9uXOUvvPBCnnnmGbZt28bFF1/Mk08+SVFREUuWLCE9\nPZ2BAwc2udRvazZs2MBdd93FokWLyM/PZ/r06W16nhrxLi1sjElNns/cg+EI6fuwtvbFF1/MnDlz\neOaZZ7jwwgspLi6mV69epKens3DhQjZu3Nji47/zne/w1FNPAbBixQqWL18OQElJCTk5OXTt2pXt\n27fzyiuv1D6muaWGjz/+eF544QUqKiooLy/n+eef5/jjj4+7LcaY1HXwZu4JEghHyE6Pv5nDhw+n\ntLSUfv360adPHy677DLOPPNMRo4cSUFBAUOGDGnx8ddeey1XXnklQ4cOZejQoYwb505yGT16NGPG\njGHIkCEMGDCAiRMn1j5mxowZTJkyhb59+7Jw4cLa7WPHjmX69OmMHz8egKuuuooxY8ZYF4wxplWe\nXvJXVVmxtYSenTPo3bVT6w/wKFvy1xjviHfJ37j6K0RkioisEZF1IjKzifvzReR5EVkuIp+IyIi2\nVDrRgmFFVfepW8YYY7yg1agnIn5gFjAVGAZcIiLDGhS7FVimqqOAHwD3JrqibVFzGbSMNAvuxpjU\nEk/UGw+sU9X1qhoA5gBnNygzDHgLQFW/AAaKyCFtqVAiu4kC0eCeypl7srrdjDHJFU/U6wdsjrld\nGN0W6zPgPAARGQ8cBvTf18pkZWWxa9euhAWkYCiauadocFdVdu3aRVaWXa3JmFSTqNkyvwPuFZFl\nwOfAp0C4YSERmQHMADj00EMbPUn//v0pLCykqKgoIZXaUxGgKhhmTWnqDqZmZWXRv/8+72eNMR1c\nPMF9CzAg5nb/6LZaqloCXAkg7rz8DcD6hk+kqg8CD4KbLdPw/vT0dAYNGhRv3Vv1/b99TEllkBev\nH9t6YWOM8ZB4+isWAYNFZJCIZADTgHmxBUQkL3ofwFXAO9GAn1Rb9lTSP98uWmyMST2tBndVDQHX\nAwuA1cBcVV0pIteIyDXRYkOBFSKyBjer5sb2qnC8VJUteyvpl5+6XTLGmNQVV5+7qs4H5jfY9kDM\n3x8CRyW2avtnZ1mA6lCE/hbcjTEpyLPTSAr3uItf9Muz4G6MST2eDe5b9rpVEq1bxhiTijwb3Av3\nRIO7Ze7GmBTk2eC+ZU8lXTul0yUrPdlVMcaYA86zwb1wT4UNphpjUpZng/uWvZXWJWOMSVmeDO6q\naicwGWNSmieD+96KIOWBsM2UMcakLE8G990VAQB6dM5opaQxxniTJ4N7aVUIgC5Znr9ErDHGNMmj\nwT0IYNMgjTEpy5PBvaTSZe65FtyNMSnKk8G9LnO3bhljTGryaHC3PndjTGrzaHAPIgI5GRbcjTGp\nyZPBvaQqROfMNHw+SXZVjDEmKTwa3IM2mGqMSWmeDO6lVSHrbzfGpDSPBnfL3I0xqS2u4C4iU0Rk\njYisE5GZTdzfVUReEpHPRGSliFyZ+KrGzzJ3Y0yqazW4i4gfmAVMBYYBl4jIsAbFrgNWqepoYDLw\nRxFJ2sIuJVVBC+7GmJQWT+Y+HlinqutVNQDMAc5uUEaBLiIiQGdgNxBKaE33QWlViNxO1i1jjEld\n8QT3fsDmmNuF0W2x/gIMBbYCnwM3qmokITXcR6pq3TLGmJSXqAHV7wHLgL7AMcBfRCS3YSERmSEi\ni0VkcVFRUYJeur7KYJhwRG3RMGNMSosnuG8BBsTc7h/dFutK4Dl11gEbgCENn0hVH1TVAlUt6Nmz\nZ1vr3CJbesAYY+IL7ouAwSIyKDpIOg2Y16DMJuAkABE5BDgaWJ/IisarpNKW+zXGmFbTW1UNicj1\nwALAD8xW1ZUick30/geAXwGPisjngAA/V9Wd7VjvZpVU1Sz3a5m7MSZ1xRUBVXU+ML/Btgdi/t4K\nnJrYqrWNXajDGGM8eIZqqWXuxhjjveBeYpm7McZ4L7jbbBljjPFkcA/i9wnZGf5kV8UYY5LGg8Hd\nnZ3qVkIwxpjU5LngXlJpi4YZY4zngntpVYgumTaYaoxJbd4M7pa5G2NSnOeCe0lV0Jb7NcakPM8F\nd8vcjTHGg8G9xK6faowx3grukYhSVm2ZuzHGeCq4lwdCqNrZqcYY46ngXrdomHXLGGNSm6eCuy0a\nZowxjqeCuy0aZowxjseCe03mbsHdGJPaPBbcazJ365YxxqQ2TwX32uundrLM3RiT2uIK7iIyRUTW\niMg6EZnZxP0/E5Fl0X8rRCQsIt0SX92WlVS6bhmbLWOMSXWtBncR8QOzgKnAMOASERkWW0ZV/6Cq\nx6jqMcAtwL9UdXd7VLglpVUh0v1CZpqnDkiMMWafxRMFxwPrVHW9qgaAOcDZLZS/BHg6EZXbV6VV\nQbpkpduFOowxKS+e4N4P2BxzuzC6rRERyQamAM/uf9X2nS0aZowxTqL7L84E3m+uS0ZEZojIYhFZ\nXFRUlOCXtkXDjDGmRjzBfQswIOZ2/+i2pkyjhS4ZVX1QVQtUtaBnz57x1zJOlrkbY4wTT3BfBAwW\nkUEikoEL4PMaFhKRrsAJwIuJrWL8XJ+7BXdjjGk1EqpqSESuBxYAfmC2qq4UkWui9z8QLXou8Jqq\nlrdbbVvhMnfrljHGmLjSXFWdD8xvsO2BBrcfBR5NVMXaoqTS+tyNMQY8dIZqOKKUB8LWLWOMMXgo\nuJfZipDGGFPLM8G9Zi1365YxxhgPBXdby90YY+p4JriXVbvg3tmCuzHGeCe4l0eDe06mBXdjjPFM\ncK/N3C24G2OMd4K7Ze7GGFPHM8G9NnPPsOBujDGeCe7l1WEAcjL9Sa6JMcYkn3eCeyBEZpqPNL9n\nmmSMMW3mmUhYVh2ywVRjjInyTHAvrw7ZYKoxxkRZcDfGGA/yTHB33TI2mGqMMeCh4F5eHbbM3Rhj\nojwU3K1bxhhjangmuJdVh+wEJmOMifJMcLfM3Rhj6sQV3EVkioisEZF1IjKzmTKTRWSZiKwUkX8l\ntpoti0QvsWcDqsYY47Sa6oqIH5gFnAIUAotEZJ6qroopkwfcB0xR1U0i0qu9KtyU8oAtGmaMMbHi\nydzHA+tUdb2qBoA5wNkNylwKPKeqmwBUdUdiq9myunVlLLgbYwzEF9z7AZtjbhdGt8U6CsgXkbdF\nZImI/KCpJxKRGSKyWEQWFxUVta3GTahZEdIusWeMMU6iBlTTgHHA6cD3gP8WkaMaFlLVB1W1QFUL\nevbsmaCXjlnL3WbLGGMMEEefO7AFGBBzu390W6xCYJeqlgPlIvIOMBr4MiG1bIVdqMMYY+qLJ3Nf\nBAwWkUEikgFMA+Y1KPMiMElE0kQkGzgOWJ3YqjbPLrFnjDH1tRoNVTUkItcDCwA/MFtVV4rINdH7\nH1DV1SLyKrAciAAPq+qK9qx4rLrZMjYV0hhjIL5uGVR1PjC/wbYHGtz+A/CHxFUtfmXR2TKWuRtj\njOOJM1Stz90YY+rzTHAXgewM65YxxhjwSHAvqw6Rk5GGiCS7KsYYc1DwRHB3i4ZZ1m6MMTU8Etzt\nQh3GGBPLE8HdXWLPgrsxxtTwRHAvj/a5G2OMcTwR3MvsQh3GGFOPJ4J7eSBkF+owxpgY3gjuNqBq\njDH1eCK4l1XZgKoxxsTq8ME9EIoQCEcsczfGmBgdPriX23K/xhjTSIcP7raWuzHGNNbhg3vdWu4W\n3I0xpkbHD+7VdqEOY4xpqMMHd7tQhzHGNNbhg7tdqMMYYxrr8MHdBlSNMaaxuIK7iEwRkTUisk5E\nZjZx/2QRKRaRZdF/v0h8VZtmmbsxxjTWakQUET8wCzgFKAQWicg8VV3VoOi7qnpGO9SxRTagaowx\njcWTuY8H1qnqelUNAHOAs9u3WvErqw6T7hcy0yy4G2NMjXiCez9gc8ztwui2hr4tIstF5BURGd7U\nE4nIDBFZLCKLi4qK2lDdxsptuV9jjGkkUQOqS4FDVXUU8GfghaYKqeqDqlqgqgU9e/ZMyAvbhTqM\nMaaxeIL7FmBAzO3+0W21VLVEVcuif88H0kWkR8Jq2QK7xJ4xxjQWT3BfBAwWkUEikgFMA+bFFhCR\n3iIi0b/HR593V6Ir25TyQMgGU40xpoFWU15VDYnI9cACwA/MVtWVInJN9P4HgAuAa0UkBFQC01RV\n27Hetcqqw+RmWeZujDGx4oqK0a6W+Q22PRDz91+AvyS2avEprw7Rt2tWMl7aGGMOWh3/DNUqmy1j\njDENdfjgXm4DqsYY00iHDu6qSnnAgrsxxjTUoYN7ZTBMRG1dGWOMaahDB/e6FSFtKqQxxsTq0MG9\nPHqhDsvcjTGmvg4e3G25X2OMaUqHDu52oQ5jjGlahw7uJZVBAHKz0pNcE2OMObh06OBeHA3uXTtZ\ncDfGmFjeCO7ZFtyNMSZWhw7ueyuC+AS6WJ+7McbU06GDe3FlkNxO6fh8kuyqGGPMQaVDB/e9lUHy\nrL/dGGMa6dDBvbgyaIOpxhjThI4d3CsCdM3OSHY1jDHmoNOxg7tl7sYY06QOHdytz90YY5oWV3AX\nkSkiskZE1onIzBbKHSsiIRG5IHFVbFokopRUBsmzOe7GGNNIq8FdRPzALGAqMAy4RESGNVPu98Br\nia5kU0qrQ0TUzk41xpimxJO5jwfWqep6VQ0Ac4Czmyj378CzwI4E1q9ZxRW29IAxxjQnnuDeD9gc\nc7swuq2WiPQDzgXuT1zVWmbryhhjTPMSNaB6D/BzVY20VEhEZojIYhFZXFRUtF8vuLcyAECeTYU0\nxphG4lmUZQswIOZ2/+i2WAXAHBEB6AGcJiIhVX0htpCqPgg8CFBQUKBtrTTUZe42oGqMMY3FE9wX\nAYNFZBAuqE8DLo0toKqDav4WkUeBfzYM7Im21/rcjTGmWa0Gd1UNicj1wALAD8xW1ZUick30/gfa\nuY5Nsj53Y4xpXlxr5arqfGB+g21NBnVVnb7/1WpdcWWQzDQfWen+A/FyxhjToXTYM1T3VgTq97fv\nXAdv/Rq2LElepYwx5iDRYa9yUbuuTMVu+NedsOghiITgnTth1MVw0m3QtV/rT2SMMR7UgTP3IEel\n74I/j4NP/gpjLod/XwqTboaVL7jtGz9MdjWNMSYpOmxwL64MMiX0JlTthasXwpn3Qvcj4OTb4PpF\nkNkFPvhTsqtpjDFJ0XGDe0WA8RX/goGToO8x9e/MPwyOuRS+XAAl3ySngsYYk0QdNrj3qVpHr8Bm\nGH5e0wXG/gA0DMuePLAVM8aYg0CHDO6BUISTIu8TwQ9Dz2y6UPcjYODxsPRxiLS4KoIxxnhOhwzu\nxRUBTvd9zLbux0JOj+YLjpsOezfC1+8csLq1m/KdUF2W7FoYYzqIDhncqzYvZaBvO9sHnNZywSFn\nQKd8WPLYgalYe3pkKjxxvh2FGGPi0iHnuaetfpGg+qk4YmrLBdOzYNQ0WPw32LsZOuUBApmd978S\nqrBzLax7Hda9Abs3wGETYfDJcPhkt1NJlD1fw84v3d+LHobjZrRet6/fdfUbdILronKLuhlz8IpE\nwNcg3ywrgi9fcb/jNFsBdl90vOCuSt6Gf/J+ZAT5eb1aLz/uCvj4frhnRN22YefAOfdDRnbb6vDN\nZ7Dgv1wABehxFPQaCl+8BMueAPFB/2PhyFNg0HcgI8eV82dAj8H7HmjX/8v932s4vHk7HD0V8gY0\nLheshGVPwScPQdHquu15h8Hwc+DE/2c/EHPwUXUnIr77Rxh2Nhz3Y+g9Ej66322rLnHlxv4gufXs\nYDpecN/6KZ3KC3k5chrXxbNoWK+hcOGjUFzobpdugw9nuWz4kjmQ26eubPku+Oot+OpNqCqBboOg\n2+HQuRcggMKaV90MnOxucOqv3YBu/mHu8eGQW/6gJptfeAcsbFCfHkfD+Kth9CXxH0Fs+Bd07g2X\nPA33fQvlBA85AAARQElEQVRevhkunVt/J6EKf7/cvW6f0XD2fTBgvHvs2tfh/XtdJn/hY60H+A3v\nwJJHYcAEGD0NsnLjq6dJjqoS99n2Hnlw7bxVobwIdq93Y0Y1cnq676aIK7PwN+7M8gHHwZpX4PO5\nkNEFAqVw1BTYvgpWPh9fcFeFN/8HDp0AR32v/dpWI1gJ696EXWtdO8t2wKl3uCQuyUR1v5ZVb7OC\nggJdvHjxvj9w/dvsfP7nfLfoZt75xbltu1jHmlfgmR+5bpoR57kulV1fQdEXgEKnbi6g7/kaQlX1\nH+vPgOOugeN/Gu3maUFZEWxZDGG3giUVO13//zfLIDPXzcU/9mrocWTzz6EKdw2Gw0+E8x9y2cyr\nM+HcB2H0xXXlPpsDz//Y7XC+dV3jo4NPHoL5/+F+LBc9DmmZjV9r5zp4/Rew5mXI6AyBMvcjGz0N\nBk50O7r8Qe6+Gg0Pow9GwSp3aH/IiIPiR5dQJVvh0TNg91fucxl0Agw7C0Ze1PJn03DspmHZUABW\nz4NDv9XyMh6qULbdBbbd693vaPd6V5/dG9x3qCk9h7gkp7gQ3vtfF7jPuBeC5e67vPF9GHsFHHEi\nvHG7S07+Yy3kdG/5/djwLjx2hnsvrnnXfWfbw56Nrrt36eNQucdty+kJgQrXDXrVm+22oxWRJapa\n0Gq5Dhfcgbtf/5I/vbmWr35zGn5fG/uSt30Ocy6F0u11GXqf0a4rpe8x4PO7H0DpVrd+TY0uvaOZ\nfBupuuz+47+6bCQShCNPhtPvrjsCiLV9Jdz/bTh7lltiIRKG2VNc19AlT7nHlhXBrGNd99CVrzb/\no170N5f1D5gAfUa5baEq90XdvQGKN7supONvhgn/5jKmT/4KK55z9WxKdnf33nU73GUxuze4H7cv\nre59HXI6jLyg6cdX7oH5P4MvXoau/V35vmNh0k+a/nFUl7ojiw3vuGx193pX75xe0XoMqqtPbj8X\n1Jc85nasGV3cUdzgk1v9mOKm6uryyYOw+WP44QL3424P4RCEq+u6+Yq3uEBWVgSn/BK2rXBHbsWb\nXTJwzv31j0xrrJoHz1/jAimA+GHIaTD+x+6kwC/+6Xbyu9dDVlc4439hxPmubOl21/W49dO6zzpY\nUffcvjTXDVj7WRzh3o/ao1/cb++TB12SA25W2+n/2/z3dtvn8MAkdxb6uOktv0f/dy58s9x9X3sc\nDVe+Av4EdFDU/G7XRo/Ktyxx3a9Dz4CCH7rvbFau+x7PuRQm3gin/M/+v24TPB3cfzlvJc8tLWT5\nL/fzsKsme0lW9lm6HZY+5rKSQ78Flz/TuMyH98GCW+CmFXX97BW74fGzoOhLmPaky3RWz4Mfvwu9\nhrT8mksfh7fugLC7TGG9H2OPo1wG1eWQ+o8JlNdlZrs3QKjabdcIlG2LZmsb3AB2tyPcjzoSduWL\n1kBJIUy4Dk79ldtp1tjwrgsyZdvcgFl1MexaDztWwtGnu0BcE+ALl8Cbv4SNH7gF4tKzXRbe7Qi3\nUyjfWZcxlsdewlHcGMXoafDOH9zOcuqdLmtsqGyHG0epaSe4IHnEd5vOGHd9BXMuc+Mbnbq5Hc+x\nV8HU37X8GbTFxg/guR+7ZGPAcS6j/fQJ15X4/edcNwe4ILTkEXj1Vvd5nHmv68eusekjeOwsOGQY\nHBWdkFCxy3WFVO5x2Wd5kcusJ93sFuQrXAQjL3RlV77gAmf3wS5o1+xIa/51HRBfMFV1z7vzSxh9\nacu/QVX4S4HbWV8xr/lyWz+FBye7RQPzDoVnfwSTb4HJM1uvT0siEXjp3937jUD/Ahh8qjvy7tq/\ncfl5N7jf2RUvwaDj67aXbnNdOOvecF1Go6e1qTqeDu43zfmUpZv28s5/npjgWiXJ+/e6TOmKl9wA\nbKynLnY/gBs+rb+9Yjc8fjbsWOWC3eRbYfLPD1yd4xUOwYJb3RHAUVNdgN/4Aax9zWU53Q533U39\nxtU9prYLaSpcMNuNkbz9W+h8CIy60B1dDTiu+cPeqhLYs8EdkfQZBfkD3fbqMnj2qrrZF6fc7o7E\nwB2d/PMmqCp2t7v0cUc1lXsAcWMrFz1ev7vrpRth+Vw47Q8us33xepfZ/XR1XXYdj0A5vHCt2xHW\n6HYEHHmSm3m17EnXdZF3mMsU17/tstnMXPj+8y7YNLRzrWvrN8vgiJPge792GfrfTnHnhvzwtfo7\nrGAlfP4MrH4JjjoVxk53QToccjvFd+50XR3HXOZ2jO11dNKct+5wg6s//RI692y6zNwr3JjZT1a4\nI47nZrg2/fDVup3fvoqE3ef62VMuG594kxtva0mgHB443n1/Rl7oEo6da6Pdvrjv8fE/dQPHbeDp\n4D79kU/YVRbgpX+flOBaJUmwymUmOT3gqrfqsphwEH4/yHVpnHlP48dV7oEnLnCZ1I/eOLgG0xr6\n+EF49ecu2wfI7R+dwXNr04Fw0cPw8k8hK88tDjfiAjj9j62Pc7QmEnY7ivfuceMnk25yWfpnT0G/\nApfV9xri6hQJw9Zl7uhq6WMw/WXXbQGuT/quwS6DO/8ht23jh/DIlPi6D2qEQ/D3y9zO7ujT3JGN\nRmDrZ1C8qa7cmO/DlN+6BfHAHfWlZbb8foQCrvvjX3e6wcmsPPf8P3rdHV3ti9Lt7j1JxDTitqjp\nnjz9bjj2R43v37nO/YYm/cQtHghuR10TZK96w2Xz+yIShhf+DZbP2fcjgC1LYPZU91nmD3RJzKET\nXDfqISP2q7cg3uCOqibl37hx47Stzpn1nl7+8EdtfvxB6dOnVG/LVf38mbptGz9y21Y81/zjIhHV\nUKD965cIX3+g+uH9qttXuXq3ZvEjqnePUP3s74mvy66vVOdc5t7fX+apvvXr5t/H6nLV3/RXfeaq\num2rX3aPXbOgblskonrft1Xvn1jXvrKdqq/eqvrhfapFa+u3OxJRnXeDe55PHq7/mpGIatGXqh/9\nVfXL1/evrWU7VV/+D/deFi7Zv+dKlkhE9c8Fqo+c3vT9L1yn+qteqqXb62/fvkr1twNU/3ysavku\nty1Qqfr+n9x729L38OWfuc/m7TvbVueqEtVwqG2PbQGwWOOIsR1vKiRQXBGkb16nZFcjsUZdBB/+\nxU3jGnKmy8I3ROe3D/xO848TAX8HuY7sYd9y/+I1bnr8GfC+6nY4XPwEbF7kuh/6jmm+bEa2O7z+\n9Ak47U53gtrnc91g8hExXYMiLqv8509g8yeu6+Kxs1zXGQrMdF0rvUe6168ucVNOJ93cOBsVcWMK\niZjdk9PddR2d9of9f65kEXGLBL5zpzuKiB0XKtvhxp3GXdF4skOvoTDtKTfQOucyN/j55v/UHRWV\nbYPv/nfj2WWf/d11JU64Dk74WdvqXHOUlSRxHRuIyBQRWSMi60Sk0bGJiJwtIstFZJmILBaRdu0v\nKa4Mkue1C2P7/HDy7W765f3fcl/ET59wgaC16V+m7QYc23JgrzHuCjdTZflcN3C65hUYfm7jHevI\ni1xf+Lt/hMfOdIOzP3gBbvwMTrvLfZ4718LHD7jAPvpSOOkX7dI0zxlxnuvmWPZE/e1LH3Ndk8dd\n0/TjBk6Ccx+ATR/Ac1e5WS3ff8FNtXz3j+7EwNju6W2fu/GUwya124yXA6HVzF1E/MAs4BSgEFgk\nIvNUdVVMsTeBeaqqIjIKmAu0Mm2jbVSVvTWX2POaI0+CKb93GfuudS4jKbgy2bUy4KbJ9jnGTavM\nzI0Oll3UuFxmZ3eC2id/hbROcOnf4fAT3H3jr66bpRMJu0Hx5gYHTWM9j3aDwx/d76bqpndyYxaL\nH3UDzy0d5Yw4HxA302vURS6ZGnSC+/+9/3VJ1YAJbjryqzPd0dmFjyRmGmWSxFPz8cA6VV0PICJz\ngLOB2uCuqrFnKuTgjkHbRXkgTDii9S+O7RUiMOEa9w9cNmFrwhw8xv7AnSfw9m/d4FxzMzC+9W8u\n+zvx1vpT4WL5/BbY2+L4m+HR091R7fir4ctX3VTbqb9v/bEjGlz7wedzA7QZOW4HsfL56PZ0Nz9+\nf85nOQjEE9z7AZtjbhcCxzUsJCLnAr8FegGnJ6R2Tdhb4eZn53U6iGeGJIoF9oPLyAvhtf/nlpE+\n/qfNfz75A+GHrxzQqqWMwya6dZs++BOMu9LNqsrt7868bgsRt1zAKb+KnivxFWT3aPms8Q4iYWfv\nqOrzqjoEOAf4VVNlRGRGtE9+cVFRUVNFWlVc6c6UzPVit4w5uGXlun52qDupxxxYIm4Aeu8mN7i6\nfiEUTN//7hMRdyR16ARPBHaIL7hvAWKXIOwf3dYkVX0HOFxEGl1FQ1UfVNUCVS3o2bNth6TFFS64\ne7Jbxhz8TrrNnVjVa2iya5K6jpoCPYfCv37vulDG2GqRTYknuC8CBovIIBHJAKYB9c4BFpEjRdwx\nqoiMBTKBXYmuLMDeaObuyQFVc/DrckjdOismOXw+d/IZuEXSGi6XYYA4+txVNSQi1wMLAD8wW1VX\nisg10fsfAM4HfiAiQaASuDg62T7hhvXJ5fazhntvnrsxJn4jzndnrY69Itk1OWh1yOUHjDEmVcW7\n/EAHWIzbGGPMvrLgbowxHmTB3RhjPMiCuzHGeJAFd2OM8SAL7sYY40EW3I0xxoMsuBtjjAcl7SQm\nESkCNrbx4T2AnQmsTkeRiu1OxTZDarY7FdsM+97uw1S11cW5khbc94eILI7nDC2vScV2p2KbITXb\nnYpthvZrt3XLGGOMB1lwN8YYD+qowf3BZFcgSVKx3anYZkjNdqdim6Gd2t0h+9yNMca0rKNm7sYY\nY1rQ4YK7iEwRkTUisk5EZia7Pu1BRAaIyEIRWSUiK0Xkxuj2biLyuoisjf6fn+y6JpqI+EXkUxH5\nZ/R2KrQ5T0SeEZEvRGS1iHwrRdr9k+j3e4WIPC0iWV5rt4jMFpEdIrIiZluzbRSRW6KxbY2IfG9/\nXrtDBXcR8QOzgKnAMOASERmW3Fq1ixDwU1UdBkwArou2cybwpqoOBt6M3vaaG4HVMbdToc33Aq9G\nLzA/Gtd+T7dbRPoBNwAFqjoCd5W3aXiv3Y8CUxpsa7KN0d/4NGB49DH3RWNem3So4A6MB9ap6npV\nDQBzgLOTXKeEU9VvVHVp9O9S3I+9H66tj0WLPQack5watg8R6Q+cDjwcs9nrbe4KfAf4G4CqBlR1\nLx5vd1Qa0ElE0oBsYCsea7eqvgPsbrC5uTaeDcxR1WpV3QCsw8W8Nulowb0fsDnmdmF0m2eJyEBg\nDPAxcIiqfhO9axvgtSsD3wP8JxCJ2eb1Ng8CioBHot1RD4tIDh5vt6puAe4CNgHfAMWq+hoeb3dU\nc21MaHzraME9pYhIZ+BZ4CZVLYm9L3oBcs9MdRKRM4AdqrqkuTJea3NUGjAWuF9VxwDlNOiK8GK7\no/3MZ+N2bn2BHBG5PLaMF9vdUHu2saMF9y3AgJjb/aPbPEdE0nGB/UlVfS66ebuI9Ine3wfYkaz6\ntYOJwFki8jWuu+27IvIE3m4zuOysUFU/jt5+Bhfsvd7uk4ENqlqkqkHgOeDbeL/d0HwbExrfOlpw\nXwQMFpFBIpKBG3yYl+Q6JZyICK4PdrWq3h1z1zzgiujfVwAvHui6tRdVvUVV+6vqQNzn+paqXo6H\n2wygqtuAzSJydHTTScAqPN5uXHfMBBHJjn7fT8KNLXm93dB8G+cB00QkU0QGAYOBT9r8Kqraof4B\npwFfAl8B/5Xs+rRTGyfhDtWWA8ui/04DuuNG19cCbwDdkl3Xdmr/ZOCf0b8932bgGGBx9PN+AchP\nkXbfDnwBrAD+D8j0WruBp3FjCkHcUdqPWmoj8F/R2LYGmLo/r21nqBpjjAd1tG4ZY4wxcbDgbowx\nHmTB3RhjPMiCuzHGeJAFd2OM8SAL7sYY40EW3I0xxoMsuBtjjAf9f/LCABprLsSrAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x20adc9e9978>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['training', 'validation'], loc = 'upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2133/2133 [==============================] - 0s     \n",
      "Test accuracy:  0.390060947037\n"
     ]
    }
   ],
   "source": [
    "cnn_rand_best_model = cnn_rand(embed_dim=embed_dim , num_tokens=num_tokens, max_len=X_train.shape[1])\n",
    "cnn_rand_best_model.load_weights('best_model.hdf5')\n",
    "cnn_rand_best_model.compile(optimizer = 'Adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])\n",
    "results = cnn_rand_best_model.evaluate(X_test, y_test)\n",
    "print('Test accuracy: ', results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CNN-static\n",
    "- A model with pre-trained vectors from ```word2vec```. All words - including the unknown ones that are randomly initialized- are kept static and only the other parameters of the model are learned"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def cnn_static(embed_dim, num_tokens, max_len, windows = [3, 4, 5], filters = 100, p = 0.5, l2 = 3):\n",
    "    inputs = Input(shape = (max_len, ))\n",
    "    embedd_intializer = Embedding(input_dim = len(word_idx) + 1, output_dim = embed_dim, \\\n",
    "                         input_length = max_len, weights = [embedding_matrix], trainable = False)\n",
    "    embedded = embedd_intializer(inputs)\n",
    "\n",
    "    \n",
    "    conv_results = []\n",
    "    for window in windows:\n",
    "        x = Conv1D(filters, window, activation = 'relu')(embedded)\n",
    "        x = Dropout(p)(x)\n",
    "        x = MaxPooling1D(max_len - window + 1)(x)\n",
    "        conv_results.append(x)\n",
    "    conv_result = concatenate(conv_results)\n",
    "    x = GlobalMaxPooling1D()(conv_result)\n",
    "    x = Dense(50)(x)\n",
    "    outputs = Dense(5, activation = 'softmax')(x)\n",
    "    m = Model(inputs = inputs, outputs = outputs)\n",
    "    m.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['acc'])\n",
    "    return m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cnn_static = cnn_static(embed_dim, num_tokens, X_train.shape[1])\n",
    "callbacks = [ModelCheckpoint(filepath = 'best_model.hdf5', monitor='val_acc', verbose=1, save_best_only = True, mode='max')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "history = cnn_static.fit(X_train, y_train, epochs = 100, callbacks = callbacks, validation_split = 0.1, batch_size = 50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4VFX6wPHvSU8gQAo1kAIESEIndBAQVEBROthREbu7\ntl109yfq6lrWggVUUEQERURUVECpIj2hSifUJBBIKCGB9JzfH2fSE5KQCZNM3s/z5Enmzp2550xm\n3nvue8oorTVCCCHsi4OtCyCEEML6JLgLIYQdkuAuhBB2SIK7EELYIQnuQghhhyS4CyGEHZLgLoQQ\ndkiCuxBC2CEJ7kIIYYecbHVgX19fHRgYaKvDCyFEtbR169YErXX90vazWXAPDAwkMjLSVocXQohq\nSSl1vCz7SVpGCCHskAR3IYSwQxLchRDCDtks516cjIwMYmJiSE1NtXVR7IabmxtNmzbF2dnZ1kUR\nQlxDZQruSqnBwPuAI/CZ1vqNQvfXBeYC/pbnfFtr/UV5CxMTE4OnpyeBgYEopcr7cFGI1pqzZ88S\nExNDUFCQrYsjhLiGSk3LKKUcgWnAECAUuF0pFVpot8eAvVrrDkB/4B2llEt5C5OamoqPj48EditR\nSuHj4yNXQkLUQGXJuXcDorTWR7TW6cB84LZC+2jAU5moXBs4B2ReTYEksFuXvJ5C1ExlCe5+QHS+\n2zGWbfl9BIQAJ4G/gL9prbOtUkIhhLAj7684xLYT5yv9ONYaLXMTsANoAnQEPlJK1Sm8k1JqklIq\nUikVGR8fb6VDW8+FCxeYPn16uR83dOhQLly4cMV9XnzxRVasWHG1RRNC2IFtJ87z3oqDrD1Y+fGv\nLME9FmiW73ZTy7b87gMWaSMKOAq0KfxEWusZWutwrXV4/fqlzp695koK7pmZV84wLVmyhHr16l1x\nn1deeYVBgwZVqHxCiOpLa81/f91HfU9XHuzbvNKPV5bgHgEEK6WCLJ2k44HFhfY5AQwEUEo1BFoD\nR6xZ0Gth8uTJHD58mI4dO9K1a1f69u3LrbfeSmio6T8ePnw4Xbp0ISwsjBkzZuQ+LjAwkISEBI4d\nO0ZISAgPPvggYWFh3HjjjaSkpAAwYcIEFi5cmLv/lClT6Ny5M+3atWP//v0AxMfHc8MNNxAWFsbE\niRMJCAggISHhGr8KQtRcxxIusXBrDJfSrqrL8Ip+33uayOPneWpQK2q5Vv4o9FKPoLXOVEo9DvyG\nGQo5S2u9Ryn1sOX+T4D/ALOVUn8BCvin1rpCUenln/ew9+TFijxFEaFN6jBlWFiJ97/xxhvs3r2b\nHTt2sGbNGm6++WZ2796dO4xw1qxZeHt7k5KSQteuXRk1ahQ+Pj4FnuPQoUN88803zJw5k7Fjx/L9\n999z1113FTmWr68v27ZtY/r06bz99tt89tlnvPzyy1x//fU8//zzLFu2jM8//9yq9RdClCwrW/Po\nvG3sPXWRl3/ew7jwZtzTMxB/H49i909KzeD3PafZFXOBO7oH0LqRZ4nPnZGVzZtL99OyQW3Ghjet\nrCoUUKbTh9Z6CbCk0LZP8v19ErjRukWzvW7duhUYH/7BBx/www8/ABAdHc2hQ4eKBPegoCA6duwI\nQJcuXTh27Fixzz1y5MjcfRYtWgTAunXrcp9/8ODBeHl5WbU+QoiSfRsRzd5TF3lqUCui4pOZveEY\nX2w4xshOfjw5MJhm3h5cSstkzYF4ftl1klX7z5CWmY2Dgq+3nOCJ64N5pH8LnB2LJkTmR0RzJOES\nn90TjlMx91eGKjVDNb8rtbCvlVq1auX+vWbNGlasWMHGjRvx8PCgf//+xY4fd3V1zf3b0dExNy1T\n0n6Ojo6l5vSFEJUr8XIGb/9+gG5B3jw5sCVKKeKGhjBj7RHmbj7OD9tj6Rrozfbo86RmZONb24Xx\nXZtxa0c/Anw8ePnnvby7/CBLd8fxzpgOhDbJG0+yOzaR95YfpFuQNwNDGlyzOsnaMvl4enqSlJRU\n7H2JiYl4eXnh4eHB/v372bRpk9WP37t3bxYsWADA77//zvnzlT9cSoia6NyldM4mp+Xefn/lIc5f\nTmfKsNDcuSGN6rrx4rBQ1j43gPHdmnEmKZVx4c2YP6kHm18YxMu3taVLgBe+tV358PZOfHp3F+KT\n0hg+bT2f/nGYrGzNgohoRn68AVcnB14b3vaazjupsi13W/Dx8aF37960bdsWd3d3GjZsmHvf4MGD\n+eSTTwgJCaF169b06NHD6sefMmUKt99+O1999RU9e/akUaNGeHqWnMcToqbSWrNoWyxnL6Xh6eZM\nPXdn+rdugLuLY4mPOX72Ej9sj2X1gXh2xZihy10DvOkb7MucjccY39WfsCZ1izyuUV03Xh3ertQy\n3RTWiK6B3jy/aBevL93P3M3HiT6XQu+WPnwwvhM+tV1LfQ5rUlrra3rAHOHh4brwl3Xs27ePkJAQ\nm5SnKkhLS8PR0REnJyc2btzII488wo4dOyr8vDX9dRX2Z0FkNP9YuKvAtj4tffny/m44OhRtHSck\np3HTe2s5dzmdjs3qMaB1A7KyNct2x3HgdBKebk6seba/VQKw1prvt8XyxtL9jA1vyjM3ti62TFdL\nKbVVax1e2n7Scq9CTpw4wdixY8nOzsbFxYWZM2faukhCVDlnk9P475J9hAd48cV9XUlOy+T3PaeZ\nsngPb/9+gH8OLjjFRmvN5O93kZSWyZIn+xLSOC8f/tQNrTgcn4yDUlZrWSulGN2lKaO7XJtRMSWR\n4F6FBAcHs337dlsXQ4hKc+FyOgdPJ9MtyLvIfRdTM6jjVvrS1K8t2celtExeH9kOTzdnPN2cubdX\nIAdOJ/HxmsN0aFqXwW0b5+7/bUQ0K/ad4f9uCS0Q2HO0qF+7YpWqoqRDVQhxzTz73U7GfrqRZbvj\nCmx/fck+Or2ynN/2xJXwSGNDVAKLtsXy0HUtCG5YsD9qyrBQOjarxzMLdvLrrlMcS7jE0YRLvPLL\nXnq39OG+XoHWrk6VJi13IcQ1sedkIiv2ncHd2ZFnFuygZYPetGzgybzNx/l07RE83Zx44pvtzH2g\ne27Lfn/cRWavP0Z6ZjbOjg6si0ogwMeDx69vWeT5XZ0c+fiuzgyftp7Hvt6Wu72OmxNvj+mAgxXz\n3tWBBHchxFXTWnPhcgank1KJS0zlVGIqsedTiDl/mbAmdZnYNyh3+N9Hq6LwdHXi+0d7ccfMzUya\ns5VnbmzNiz/toX/r+vxvdAfGzdjIxC8j+PL+bizbHcdn647i7uxIPQ9nMrM0Lk4OvDWqPW7OxY+K\naVzXndXP9udAXBKHziRz+Ewy17WqT+O67tfyZakSJLgLIa5KWmYWd3+2hS3HzhXY7uig8PJw4ccd\nJ1EKJvZtzsHTSSzdHccT17ekVUNPpt/ZmTtmbuKxr7fRppEnH93RmdquTsy5vxujPt7AiOkbABjf\ntRn/HNwGr1pl/+4fDxcnOvl70cm/Zs/wlpx7BdSubTpiTp48yejRo4vdp3///hQe8lnY1KlTuXz5\ncu7tsiwhLERlysjK5tuIE0Sfu1ziPm//doAtx87xxPUtmXZHZxY+3JN1/xzAgf8MZssLAxnSthGv\nLdnHb3vi+GhVFLVcHLm/t1nOo1uQN/8d0Y72Tesya0JXalsW0mrq5cGc+7tzS/vGLHy4J2+Mal+u\nwC7ySMvdCpo0aZK74uPVmDp1KnfddRceHmaBoiVLlpTyCCEqT/S5yzw5fzvbT1zAt7Yrs+/rSlu/\ngpN71h6MZ+afR7m7RwDP3Ni62Od5d2xHTs7cxN/mbyc9M5sHr2teIFCP7dqMsV2bFXlca0tLXlSM\ntNzzmTx5MtOmTcu9/dJLL/Hqq68ycODA3OV5f/rppyKPO3bsGG3btgUgJSWF8ePHExISwogRIwqs\nLfPII48QHh5OWFgYU6ZMAcxiZCdPnmTAgAEMGDAAyFtCGODdd9+lbdu2tG3blqlTp+Yer6SlhYWo\niGW7TzH0gz+JOp3M/90SioujYvyMTWyIylvk9WxyGs98t5PgBrX5180lT45zd3Fk5j1d8KnliouT\nwzVZw1zkqbot96WTIe4v6z5no3Yw5I0S7x43bhx///vfeeyxxwBYsGABv/32G08++SR16tQhISGB\nHj16cOutt5a4RsTHH3+Mh4cH+/btY9euXXTunNcCee211/D29iYrK4uBAweya9cunnzySd59911W\nr16Nr69vgefaunUrX3zxBZs3b0ZrTffu3enXrx9eXl5lXlpYiLKKPHaOh+duo0PTunx4e2f8fTwY\n2q4R987awr1fbKFHc7MCauz5FBJTMphzf7cSOzZzNPB044dHe5GQnI7vNZ5+X9NJyz2fTp06cebM\nGU6ePMnOnTvx8vKiUaNGvPDCC7Rv355BgwYRGxvL6dOnS3yOtWvX5gbZ9u3b0759+9z7FixYQOfO\nnenUqRN79uxh7969VyzPunXrGDFiBLVq1aJ27dqMHDmSP//8Eyj70sKiettwOKHU79s8fvYS66Mq\n9qUu2dmaV37ZS8M6rnwzqUfuGuaN67rz3UO9GNK2MclpmSSnZVLPw5m3x3QodkJQcRrUcSuwSqK4\nNqpuy/0KLezKNGbMGBYuXEhcXBzjxo1j3rx5xMfHs3XrVpydnQkMDCx2qd/SHD16lLfffpuIiAi8\nvLyYMGHCVT1PjrIuLSyqr/ikNCZ+GYnW8P0jvYoNkPvjLnLHzM1cuJzOiqf70byE2ZbJaZlsiEog\nM9usJeXh4kjf4Pq5a578uCOWXTGJvDOmAx4uBcNCXQ9nPri9k5VrJyqbtNwLGTduHPPnz2fhwoWM\nGTOGxMREGjRogLOzM6tXr+b48eNXfPx1113H119/DcDu3bvZtcssbnTx4kVq1apF3bp1OX36NEuX\nLs19TElLDfft25cff/yRy5cvc+nSJX744Qf69u1rxdqKqmz6mijSMrOp5erEQ3MjOX8pvcD9OYHd\nxdEBVydH3l95qMhzXEzN4KNVh+jz5iomfbWVR+dt49F525jwRQST5kSSlJrB5fRM3lp2gPZN6zKi\nk9+1qp6oZFW35W4jYWFhJCUl4efnR+PGjbnzzjsZNmwY7dq1Izw8nDZtinzvdwGPPPII9913HyEh\nIYSEhNClSxcAOnToQKdOnWjTpg3NmjWjd+/euY+ZNGkSgwcPpkmTJqxevTp3e+fOnZkwYQLdunUD\nYOLEiXTq1ElSMDVA7IUU5m06wZguTRnbtRnjP93Ek/O388WErqRmZhNx9BzPfLcTZ0fF/Ek9mB8R\nzadrD/PYADOOHGDF3tM8vWAHF1MzGdimAQ/0DcKnlrni23A4gVd/3ceI6RvoGuhN3MVUPryjU42b\nxWnPZMnfGkBe1+rnHwt38uP2k6x5rj9N6rkzf8sJJi/6C7967pxKTCFbQ8M6rsyf1JMg31qcv5RO\n37dW0zfYl4/v6sLGw2e594sttG7oyX9HtKNd06LrlG84nMCj87Zx4XIGN7drzLQ7ZfhhdSBL/gpR\nTR2OT2bh1hgm9AqiST0zbX58N39iL6SwI/oCozr70TnAi/BA79zJP161XLi/TxAfrDzEgohoXvll\nL/7eHsy5v1uJk4B6tfBl8WN9mPGnafEL+yLBXQgbycrW7DmZyLqoBDYePktKehbuLo7EXkjBzdmR\nRwe0KLB/SZOFcjzQJ4jZ64/yj+934VfPna8eKDmw5/D38SjTtwyJ6qfKBXet9TX9nkF7Z6u0W02X\nkZVNemY2mVkaFNR1L7hO+V8xiTw8dyuxF8wopzaNPPGu5UJSaiYujg5MHtKm3OPC67o78+xNrZn5\n5xFm39etRi6WJfJUqeDu5ubG2bNn8fHxkQBvBVprzp49i5ubm62LUmNorZn55xHe/v0g6ZnZudtv\nCmvIv4aG4u/jwfK9p3nym+1413Jh6riO9G7pS31P60zwuadnIHf3CJDPj6hawb1p06bExMQQHx9v\n66LYDTc3N5o2te3XfdUUiZczeOa7nazYd5pBIQ3oGuiNk6MDCclpfLnhGIPe/YMbQhuyZPcp2vvV\nZea94TTwtP6JVwK7gCoW3J2dnQkKCrJ1MYQoF601y/ee5pVf9nL6YipThoUyoVdggSA7oVcgby7b\nz6JtsdwY2pD3x3fC3eXKU/eFqIgqNRRSiOpEa82ag/G8t/wgu2ISCfKtxbtjO1xxHfEzSan41nKV\n8eTiqslQSCEqUVpmFi8s2s3322Jo6uXOW6PbM7KTH06OV570XRlpGCGKI8FdiHI6m5zGw3O3EnHs\nPE8ODObxAS1xcZKVPETVUqZ3pFJqsFLqgFIqSik1uZj7n1NK7bD87FZKZSmlvK1fXCFs60BcEsOn\nr2dXTCIf3t6Jp29oJYFdVEmlviuVUo7ANGAIEArcrpQKzb+P1vp/WuuOWuuOwPPAH1rrc0WfTYjq\n66cdsQyftp7UjGy+fagnwzo0sXWRhChRWdIy3YAorfURAKXUfOA2oKTFyG8HvrFO8YS4dtIyszib\nnI6nm1PutP60zGwupmQwfc1hZm84RtdAL6bd0ZkGdSR3Lqq2sgR3PyA63+0YoHtxOyqlPIDBwOMl\n3D8JmATg7+9froIKUVmyszU/7YzlzaUHiLto1th3UODooMjIyhtNdn/vIJ4f2gbnUjpNhagKrN2h\nOgxYX1JKRms9A5gBZiiklY8tRKlyZpDuP5VEgzpu1Pd05ZddJ9l+4gLt/Ory2PUtSU3P4mJqBhlZ\nmjruTni6OdOqQW26W75mTojqoCzBPRbI/xXlTS3bijMeScmIKuy95Qf5YFUU9T1duXA5nYwsTX1P\nV/43uj2jOjeV8efCbpQluEcAwUqpIExQHw/cUXgnpVRdoB8g39IsqqRZ647ywaooxoU3441R7dAa\nzl9Op7abE65OMltU2JdSg7vWOlMp9TjwG+AIzNJa71FKPWy5/xPLriOA37XWlyqttEKUw9qD8cSc\nT8HZURGXmMo7yw8yOKwRr41oi1IKpcCnnCsvClFdyPIDwi59uPIQ7yw/WGBb75Y+fH5vV9ycpZUu\nqi9ZfkDYrbPJaczZeJxhHRrTsoFngfu01rz9+wGmrT7MyE5+PDe4NZlZmqxsjb+3h+TURY0hwV1U\nK/tOXeTBOZHEnE9h2uoo7u8TxJMDg3F1cmD/qSS+iTjB15tPcHs3f14b3laCuaixJLiLKmXF3tMs\n3nmS525qTTNvjwL3Ldsdx9MLduDp5sTs+7qy9K84Zqw9woLIaNIysknJyALMePT/uyVE1jUXNZoE\nd1ElaK35dO0R3ly2H61h9YEzvDmqPUPbNSbqTBLvrTjEr7tO0aFZPWbe3YUGddzo37oB47s147M/\nj1Lf05UuAV50CfDK/VJpIWoy6VAVNpeemc2/fviL77bGcHP7xvx9YDDPLtzFzugLdAnwYvuJ87g7\nO3Jf7yAev76ldIiKGk06VEW18b/f9vPd1hj+NjCYvw8KRinFdw/15O3fD/DNlhM82Lc5D/VrgXct\nF1sXVYhqQ4K7sKkzF1OZs/E4o7s05akbWuVud3Fy4IWhITw/pI3kzoW4CrICkrCpj/84TGa25onr\nWxZ7vwR2Ia6OBHdhM6cvpjJv8wlGdfYjwKeWrYsjhF2R4C5s5uM1h8nO1jw+INjWRRHC7khwFzYR\nl5jK11tOMKpzU/x9PEp/gBCiXCS4i2vuSHwyj87balrtJeTahRAVI6NlxDWTla35fN0R3vn9IK5O\nDrwztkORWahCCOuQ4C4q1YXL6Szfe5r1UQmsizpLQnIaN4Q25NXhbWko30MqRKWR4C4qTGvNjLVH\naFzPnVs7NMndHnUmmbs/38ypxFR8a7vQu6UvQ9s15sbQhjLEUYhKJsFdVIjWmtd+3cdn644CsP5Q\nAi/fFkbUmWTunbXFzDZ9uCfhAV4S0IW4hiS4iwp5b/lBPlt3lHt7BuDp5sxHq6PYHn2eU4mpeLo6\nMXdid5rXr23rYgpR40hwF1clITmNL9YfZdrqw4wLb8aUYWE4OCi6BHrx9Lc78K3tytyJ3fGTFRqF\nsAkJ7qLMtNbM3XScn3acZOuJ82gNIzr58d+R7XK/FGNA6was/ccAnB0dZPVGIWxIgrsoE601Uxbv\nYc7G44Q0rsOT1wdzQ2hDwprUKZJL93RztlEphRA5JLiLUmmteckS2B/sG8QLQ+VbjoSo6mSGqrgi\nrTUv/7yXLzceZ2IfCexCVBcS3MUVTV9zmNkbjvFAnyD+dbMEdiGqCwnuokS/7DrJ/347wG0dm/Bv\nCexCVCsS3EWxtp04z9MLdhIe4MWbo9pLYBeimpEOVQFAWmYWC7fGEHUmmZjzKWw+cpZGddyYcU+4\nDGkUohqS4C7QWvP8or9YtC0WDxdHmnq506O5D5OHtJEvpRaimipTcFdKDQbeBxyBz7TWbxSzT39g\nKuAMJGit+1mxnKISzfzzCIu2xfLkwGCeGhQsKRgh7ECpwV0p5QhMA24AYoAIpdRirfXefPvUA6YD\ng7XWJ5RSDSqrwMK6Vu8/w+tL9zO0XSP+PlACuxD2oiwdqt2AKK31Ea11OjAfuK3QPncAi7TWJwC0\n1mesW0xRGaLOJPPEN9sJbVyHt8d0yF1CQAhR/ZUluPsB0flux1i25dcK8FJKrVFKbVVK3WOtAorK\ncTk9k0fmbsXVyYGZ94Tj4SLdL0LYE2t9op2ALsBAwB3YqJTapLU+mH8npdQkYBKAv7+/lQ4tyktr\nzb9+2E1UfDJf3d+dJrJyoxB2pywt91igWb7bTS3b8osBftNaX9JaJwBrgQ6Fn0hrPUNrHa61Dq9f\nv/7VlllU0Ddbovlheyx/H9iKPsG+ti6OEKISlCW4RwDBSqkgpZQLMB5YXGifn4A+SiknpZQH0B3Y\nZ92iiorSWrNsdxwv/byHvsG+PHF9S1sXSQhRSUpNy2itM5VSjwO/YYZCztJa71FKPWy5/xOt9T6l\n1DJgF5CNGS65uzILLoo6k5TKhcsZtGroWeS+/XEX+c8ve1kfdZbWDT2ZOq6jdKAKYceU1tomBw4P\nD9eRkZE2Oba9mvDFFrYeP8+m5wdSyzXvvP3rrlM88c02PN2ceWpQMHf2CMDZUVaeEKI6Ukpt1VqH\nl7afDJGwE6cSU/jjYDxaww/bY7mrRwAA2dmad5cfoFVDT755sAdeMuNUiBpBmm92YtG2WLSGZt7u\nzNl4jJwrsj8OxnM4/hIP92shgV2IGkSCux3QWvNdZDTdg7x5YkAwB08ns/noOcAsLdCojhs3t29s\n41IKIa4lCe52IOLYeY6dvczY8Gbc2rEJ9TycmbPxGHtOJrLh8Fkm9A6UHLsQNYzk3O3Agshoars6\nMaRdI9ycHRkb3ozP1x0lKTUTDxdHbu8mE8aEqGmkOVfNJadl8uuuU9zSvnHuEgJ3dQ8gW2v+PJTA\n2PBm1HV3tnEphRDXmgT3au7XXSdJychiTHjeJGJ/Hw+ub90ABwUP9AmyYemEELYiaZlq6lJaJp/8\ncZhP1x6hdUNPOvvXK3D/K8Pbcuh0Es28PWxUQiGELUlwr0a01hyOv8QfB+OZsfYwpy+mcVvHJkwe\n0qbIOux+9dzxkwXBhKixJLhXE5/+cZjZG45xKjEVgA7N6jH9zs50CfC2ccmEEFWRBPdq4ODpJN5Y\ntp+ugd48cX0wfVr64u8j6RYhRMkkuFcDH6w8hIezI5/e1UVmmQohykRGy1Rxh04n8etfp7i3V6AE\ndiFEmUlwr+I+WBWFh7MjE/s2t3VRhBDViAT3KmZXzAUOxCWhtSbqTBK/7DrJPb0C8ZZWuxCiHCTn\nXoWcv5TOqI83kJGlqe/piqerE+7OjjworXYhRDlJy70KWXPwDBlZmr8NDKZXCx8up2fx2ICW0moX\nQpSbtNyrkBV7z1Df05W/DQyWr8ATQlSItNyriPTMbP44GM+gkAYS2IUQFSbBvYrYfPQsyWmZDGzT\n0NZFEULYAQnuVcSKvadxc3agd0tfWxdFCGEHJLhXAVprVuw7Q5+Wvri7ONq6OEIIOyDBvQrYH5dE\n7IUUBoVISkYIYR0S3KuAlftOA3B9mwY2LokQwl5IcK8Clu87Q4dm9WhQx83WRRFC2AkJ7jb2V0wi\nO6MvcEOItNqFENYjwd2G0jOzeW7hThrWceXunoG2Lo4Qwo6UKbgrpQYrpQ4opaKUUpOLub+/UipR\nKbXD8vOi9Ytqf6atjmJ/XBL/HdGOuu7Oti6OEMKOlLr8gFLKEZgG3ADEABFKqcVa672Fdv1Ta31L\nJZTRLu09eZFpq6MY3rEJA2WUjBDCysrScu8GRGmtj2it04H5wG2VWyz7duZiKs9+t5N6Hi5MGRZm\n6+IIIexQWRYO8wOi892OAboXs18vpdQuIBZ4Vmu9xwrlsytnk9P45I/DzNl4nKxszcfytXlCiEpi\nrVUhtwH+WutkpdRQ4EcguPBOSqlJwCQAf39/Kx26eth05CwPzI4gJSOL4Z38+NvAYAJ8atm6WEII\nO1WW4B4LNMt3u6llWy6t9cV8fy9RSk1XSvlqrRMK7TcDmAEQHh6ur7rU1cy+Uxd58MtIGtdz55O7\nutCyQW1bF0kIYefKknOPAIKVUkFKKRdgPLA4/w5KqUZKKWX5u5vlec9au7DVUfS5y9w7awu1XJ2Y\nc383CexCiGui1Ja71jpTKfU48BvgCMzSWu9RSj1suf8TYDTwiFIqE0gBxmuta0zLvCQnL6Rw76wt\npGZksfCRXjSp527rIgkhaghlqxgcHh6uIyMjbXLsyqa1ZvHOk/z7x91kZ2tm39+NroHeti6WEMIO\nKKW2aq3DS9tPvmbPyi6lZfLP73fxy65TdAnw4r2xHfH38bB1sYQQNYwEdyv7eM1hfv3rFM/d1JqH\n+7XAUb4yTwhhAxLcrehyeiZfbTrODSENeWxAS1sXRwhRg8nCYVa0ICKaxJQMHurX3NZFEULUcBLc\nrSQzK5vP1h2lS4AXXQKk81QIYVsS3K1k2Z44Ys6nMOk6abULIWxPgvtVyszK5uSFFLKyNVprZqw9\nQpBvLfkeVCFElSAdqlfp/ZWH+HBVFK5ODgT4eHDwdDKvjWgro2OEEFWCBPerkJWt+S4yhg5N69It\nyJuDp5MIOYCWAAAgAElEQVRpWMeNUZ2b2rpoQggBSHC/KhsPnyXuYir/viWEW9o3sXVxhBCiCMm5\nX4VF22PwdHWS/LoQosqS4F5Ol9MzWbY7jpvbN8bN2dHWxRFCiGJJcC+n3/ec5nJ6FiM6+dm6KEII\nUSIJ7uW0aHssfvXcZZVHIUSVJsG9HM5cTGXdoXhGdPLDQYY8CiGqMAnu5TB303GyNYzoLCkZIUTV\nJkMhy0BrzYerovhgVRRD2jaiRX35qjwhRNUmwb0UWdmalxbv4atNxxnZ2Y83R7W3dZGEEKJUEtyv\nQGvN84t2sSAyhof6NWfy4DZYvgdcCCGqNAnuV/BtRDQLImN4fEBLnr2pta2LI4QQZSYdqiXYczKR\nFxfvoW+wL0/d0MrWxRFCiHKR4F6Mi6kZPDpvG94eLkwd11FWehRCVDuSlinGlJ/2EHM+hW8n9cCn\ntqutiyOEEOUmLfdCdkZf4IftsTzSrwXhMgtVCFFNSXDPR2vN60v34VPLhYf7t7B1cYQQ4qpJcM9n\nzcF4Nh05x5MDg6ntKhkrIUT1JcHdIitb8+bS/QT4eHB7N39bF0cIISpEgrvFD9tj2R+XxLM3tsbF\nSV4WIUT1VqYoppQarJQ6oJSKUkpNvsJ+XZVSmUqp0dYrYuVLz8zmveUHad+0Lje3a2zr4gghRIWV\nGtyVUo7ANGAIEArcrpQKLWG/N4HfrV3Iyvbd1mhiL6Tw9A2tZClfIYRdKEvLvRsQpbU+orVOB+YD\ntxWz3xPA98AZK5av0qVlZjFtVRSd/OvRr1V9WxdHCCGsoizB3Q+Iznc7xrItl1LKDxgBfHylJ1JK\nTVJKRSqlIuPj48tb1kqxIDKGk4mpPDWolSwKJoSwG9bqOZwK/FNrnX2lnbTWM7TW4Vrr8Pr1bd9K\nTsvMYvrqKDr716NvsK+tiyOEEFZTlsHcsUCzfLebWrblFw7Mt7R8fYGhSqlMrfWPVillJVkQEc2p\nxFTeGt1eWu1CCLtSluAeAQQrpYIwQX08cEf+HbTWQTl/K6VmA79U5cCekZXNlxuO8d7yg4QHeNGn\npbTahRD2pdTgrrXOVEo9DvwGOAKztNZ7lFIPW+7/pJLLaFWbjpzlxZ92c/B0Mv1a1ee1EW2l1S6E\nsDtlmmOvtV4CLCm0rdigrrWeUPFiWV9mVjZTVxxi2poomnq5M/OecAaFNJDALoSwSzViAZUzSak8\n+c12Nh05x7jwZrx0axjuLo62LpYQQlQauw/u0ecuM/qTDSSmZPD2mA6M7tLU1kUSQohKZ9fB/WJq\nBvfPjiAlPYtFj/QmtEkdWxdJCCGuCbsN7hlZ2Tw2bxtHEy4x5/5uEtiFEDWK3Qb3F3/aw5+HEnhr\ndHt6yVBHIUQNY5dr2+6MvsA3W07wUL/mjA1vVvoDhBDCzthlcP9+WwyuTg48NqClrYsihBA2YXfB\nPT0zm8U7T3JjWCPquDnbujhCCGETdhfcV+0/w4XLGYzs7Ff6zkIIYafsLrh/vy2G+p6u9JVOVCFE\nDWZXwf3cpXRW7z/D8I5NcHK0q6oJIUS52FUEXLwjlsxszSiZhSqEqOHsKrgv2h5LaOM6tGkkE5aE\nEDWb3QT3Q6eT2BWTeO06UrOzIS3p2hxLCCHKyW6C+3dbY3ByUAzvVEpw3/ENxB+o+AE3fgTvhcH5\n4xV/LiGEsDK7CO4ZWdks2hbLgDYN8K3tWvKOWZnw06Ow9n8VP+jJ7ZCaCD89ZlrxQghRhdhFcF97\nMJ6E5DTGlNaRmnQKdDYcWw9aV+yg5w6Dax049idEzKzYcwkhhJXZRXD/LjIG39ouDGjTIG/j5XNF\nA/jFk+Z30kk4f7RsT56Ralro+WkNZw9Dh9sh+EZYPsXcFkKIKqLaB/dzl9JZuf80wzv64Zwztj0p\nDt4Ngb0/Fdz5Ymze38fWl/7kmWnw5S0w57aC25PPQHoy+LSEYR+Akyv8+EjFrwaEEMJKqn1w/2lH\nLBlZmtHh+VIy0ZshMxXi/iq4c05wd/GE42UI7r+9ADERcGqnacHnOBtlfvs0hzqNYcC/zDGt0VEr\nhBBWUL2D++k9XLdqJAMbpRYc2x67zfy+cKLg/hdPgnMtaDGg9Jb7zm8h4jNo1M7k6eP35913zpKC\n8bGsOtlyoPl9YuPV10UIIayoWgf35J8n0yLrCA812l/wjpMlBfdYqOsHgX0g8UTR+3Oc3gM//w0C\nesPIzyzbdufdfzYKHF2grmWteO/mUKuB/Qb3pDhJOYmqLTkesjJsXYoqpfoG98OrqR2zFoD2mXvy\ntmdnw8kd5u8LhcagXzwJdZqYoA3Ft95TE+Hbu8GtDoz+AnyDwcndBPwcZw+DVxA4OJrbSkFAT/sL\n7pnp8Nu/4J3WsHW2rUsjRPEy0+HDLrD5E1uXpEqpnsE9OxuWv0iCU0NWOvbB7eSmvJbl2ShIuwj1\n/M3Qx8y0vMclxkIdP2gQCu5ecHxdwefVGn58FM4fgzGzwbOhCeANQwu13A/npWRy+Pc0VwKJsVSK\no2vh1K6y73/uCKx+HVa9Zn62flm+4507CrNuMpO1nGvBzvnle7wQ10piNKQlmn4vkat6fofq7u8h\nbhdT1ZO0beQGp9aZnHiDkLyUTOhw2PABJMaATwszgSk5zrTcHRzAv1fRlvuGD2D/L3DTfyGgV972\nhmGw7xcT/LU2gTN4UMHH+vcwv09shHajrVvftCT4ery5Qrj3Z/DrfOX90y/DvDGWjl8FWE58DUKh\nWdfSj3fhBHzazzx07FeQcABWvWpey7qyKJuoYs4fM7/zX12Lathyz0yDVa+Q5hvGvJRu1GrV32w/\nZmmFx241Lc3gG8ztnNRM8mnTMVrHsjxBYB8z1j2npX1kDax4CUJvgx6PFjxmw7aQcs7kni/GQFZa\n0ZZ7w3bgUhtObLJyhYHdiyDjksnzzxtT+pj6Vf8xgf2en+ClC/B8DLjVhY0flu14G6eb401cBaG3\nQthIs33PjxWrR1lkZ1Xe1U9x1n8Aa9++dscT5XfuyJXvz/mMnzsKacmVX55qovoF953z4cIJNjZ/\nEo0DIaHtTcDOGdoYuw2adDI5cchb+yVnGGRucLfk3Ve9Cp/fZMaye7eAWz8yLeT8GrY1v0/vyQus\n3i0K7uPoBE27FgzuSXFmglPhSVBXkppYdDmD7V9B/TbwwHJAw1cj4Mx+EwQTYyEjJW/fo3/CpunQ\n9UFo3t9sc/WE8Pth38+lf1BSzsO2OdB2NPhaTmA+LaBRe9izqGx1yMo0P+WVnQUL7oGp7fJGPFWm\n5HhY+Yo5GW6fV/nHs6bsrJrRgXh8I3zQyVw5lyR3fScNZ/ZVTjni/oIfHjYNgYO/mbkuVVyZgrtS\narBS6oBSKkopNbmY+29TSu1SSu1QSkUqpfpYv6gWHe+AcfP4OTkE71outGhQ23SQHltnOlbi/gK/\nTuDZGByc8kbE5Ab3JuZ3w7Ym777zaxPQBr0E9y8zHamFNQw1v0//lW+MezFfvh3Qy+TmUy6Y20ue\nhfVTYeO0kusTf8DkxueNgXfawBv+8Pu/8+4/s9+Mte90twm2d3wHl+Jhend4L9T8vNXCvPEOLTdr\n53gFwQ0vFzxOt4dAOcKmj6/06kLkF6bV3uvxgtvbjjRXRWVZKG3uSPj2rtL3y09r+PUZkxZzdDGv\nXXnX7NHanJjKOlt425eQnWFOXL8+XXReRFW25Dn4/EZbl8J6Lp2FLTOL/s8PLjO/r9RZeuG4uVqH\ngn1j1nL5HHxzh0kHr/oPfD3WNEBO7bT+sayo1OCulHIEpgFDgFDgdqVUaKHdVgIdtNYdgfuBz6xd\n0FyOzhByC5HHzxEe4IVSyrTCL8WbGalZadCks2lJ1/HLF9wtSw/UtbTcHRzh7h/hwdXw2Gbo8xTU\nKuGr+dy9oE5T03I/d8S8kTwbFd3PvwegTTA+tMK0lN3qwaZP8gI+mCC0dTbMGADTusHat0wLPOg6\nCL4JNk0zHahgWu0OztBhvLndtAs8uMrMjB32AQx7H9qNgv2/wrzRcCEaRnwCLrUKlq1OY2g3BrbP\nNW/W4mSmw+ZPofkAM74/v7AR5veeH4p/bI70S+ZEe3ApHF595X3z++NN2PoF9P67qVPsVtg+p+yP\nB3NyW/wEfH6DWdjtSrIyzYmseX+463vzP/727oL/p6oqK9MEmpPb8t7X1d2m6eaEfmJDwe2HVwHK\nrOF0Zn+xD+X8cdOX5OJp/bx7djYsetD0192/DCafgAlLwNkDlr9o3WNZWVla7t2AKK31Ea11OjAf\nKDAfX2udrHXuQOha5PbgVY4zF1M5fvYy3YK8zYYAy4XChg/Mb78u5rdXQF5wT4w1/xC3enlP1KSj\n6ZwsnIYpTqO2lrRMlJmZWtxj/MLN1cKRNbD0OdO6v+t705O/+dO8/dZPNePos9JN5+3T++HRDTBy\nBoz5woyb//ExE4R3fgOthxQ88TQIgS73Wn4mwK0fwrMHYfQsGDsnr3O3sF6PQ8ZliJxlTjAXTkB0\nRN7l/e6F5k1cuNUO4BVoTpqlpWZObgedZU5IK6aUrfUdOQvWvA4d7jBXUO3HmquxFS+VfCIqTGv4\n4w1zEnauBbOH5Z0gi3Nwmek/6ToRajeAMV+aURcL7jatyKpi1WumXyC/6M2QajkJHf3z2pepMuz7\nueBvgEsJELcLuj9sruYiSmgzXjhu3p8Nw6zfcl/7FkStgMFvmLjiVtc0Jq97znzOD68q/nHpl0z6\ntDL64MqoLMHdD4jOdzvGsq0ApdQIpdR+4FdM670IpdQkS9omMj4+/mrKC8CWY+YDHx5oCe4+LaB2\nQ/NG8PAxwyDB/L6QL+dep0nZAnlxGoZBwkGT0yucb8/h4gGNO5rUx7kjMOQtaBoOrYea1njqRRNw\nVr5iOikfXgc9HzNDLnOfoxYM/8QEni+GwuWz0Pme0svn7A5tR5kO0CvVocVA+PMdeDPAXFp+Psik\ng5ZOhnVTzYiaFgOLf3zbkeZSdPtc+ON/sPCBvI7sHNFbzO8bXzX7lnYy2PezSccE3wi3fmD+P0rB\n0LfN67Xy5Ss/Psfhlaa1f92z8MBv5gpt7qiSg1/ETHMiaDXE3Pbvbk6SJzbBJ33g+IbiH1dYZX5h\nS2a6Sen98WbBjsKDS83J07XulU9gheVfQsNWsrOKdnrGHzQjshxd80algQmeYK44w0aa/rbCr3da\nsvmM1AuwBPc95Z9wl5VRtExZmSZNtOYNs0BgeKGQ1vUBE1+Wl9CAObbeBP6fHjf/RxuwWoeq1voH\nrXUbYDjwnxL2maG1Dtdah9evX/+qjxV57Dzuzo6ENbHkx5XKm5jUJF9LvF6AGSWTkWKZwFSBb2lq\nGAbZmaZ1V1y+PYd/D9NyDb0tb1mCfv8wHaUrX4GF94NPsAkkJZ1o/LtDrycgfp8pc4vrr77chV3/\nb3PCCRsJN79jWvsBvSDyc/MB6/VEyeUKGwEos4b96ldh74/w57sF94neYurXbZJJ7ax8peBcg/yO\nrTcnCL8uZl6Bo3PefQ1DoccjJn216rUrd9BqDWveNMG6453mJH7fUnOiL66PIeGQCRzhE0z6LkfH\nO0yntbMbzL7ZdJ5dKVDs+Br+F2w6zvPLSDEn0NSLJT+2LE5uM/0f6cnmtc5xYBkE9TU/R/8oWzA7\ntBzeaFZ0Mb1rbfVr8H77gumv/ZbWer/nTKMmJ6V2eLW50m7SEbo9COlJRedb5DTevCzBPe2i+YyW\nx0+PwZuBMP9Oc3I5sAw+6W3SRIF94OZ3i34mnFzh+v8zDcrdC4s+5/F1gIKzh0zDzgbKEtxjgWb5\nbje1bCuW1not0FwpVUICu+K2HD1HJ/96eatAgvknQF5KBkxwBzM+u8LBPV8O+krBPXS42fem/+Zt\na9LJ5NIjZpoP/rivwLX2lY/X/wUI6mdy0DkzYa3Br7MZKz9sqklJtB1lyvPMAZNCaj++5MfWbWry\njvf/ZoZXdn/Y5EJzWlNam5RBs+5mLsGgl82HL3JW0ec6vRe+ud18KO9YULSPAMyJqOOd5tL4y2El\nD5E8+gfEbIG+T4GTi9nm4Q1tboYjq4u2WCM+Ny3fzvcWfa4mHeGhtebkt+o/5qoiO6v4426ZAZkp\neS3MHHsXm5PayleKf1xZHV0LKLPMxbavzLaEKBMwWg0x/QWJ0XnjvK9k32KTBvx+ou1SOemXTWrl\n8tmCacp9P5vPbfgDptN/38/mvXR4FTTvZ97/fl3MVXHEZwVPZjkd/PUC8/qJisu7xx80fVw/PFJw\n+/lj8Nd30LiDaZh8eyd8M868VmO/Mp8VF4/i69N2tDnmqv8UbcAcW28+B61vhj/eMn1h11hZgnsE\nEKyUClJKuQDjgcX5d1BKtVTKnNqUUp0BV6BSEpcXUzPYH3eRrjkpmRwtB5kOlZb5Ugo56ZlzR8xs\n1ZyRMlfDuzk4uZm/fUpIy4Dp2HlkXdHJPgOeNy3J26ZB/dalH8/ZDe5dDN0nXX2Zy8PD27yGDqW8\nJfx7mB9XT2g12HwIcjpOzx428wGadTO3Ww40AeiPtwoOB9UaFj9u6njXInPs4ji7w/DpMGKGSfF8\n1NWkkqa2g/c7wtzRJi+/fAp4NjEjivJrNcT0MeRPXWSkmBFSobeZXHtxXD1h1GfmxBr5uQmIhS+t\n43bntTALp0aOWF6PyM/zlsK4GkfXQuP25iQcvckEqINLzX2tB5sO+OKOX9JzBfY17+P5d9hmZNDu\nheZ94N0iL015Idq8jiHDzPsgsI8ZMZVw0HzvQvMB5rFKmdZ7/P6Cy3zknNi8AkxfFJj/TQ6tzTDX\nGf3Me2jn1wXz4Js+NieUcV/B0/tMQ2PkTHh0s0lxXimN6+AAA6eYvqv9+YZqpiWbOgX2hsGvmzL8\n9oK5LznenLQqa8hm/uKVtoPWOhN4HPgN2Acs0FrvUUo9rJR62LLbKGC3UmoHZmTNuHwdrFa17fh5\nsjV5nak5vALghZi8wAJ5wT0mwqRK6lag5e7oZMaaw5Vb7iVp0gmejYKw4VdfhqrGv4e5bM4ZrpYz\n/btZ97x9Br1sAv769/O27f3R5McHToF6+S8KS9BhnGlN53S0BvQ2r2fSKdjwIZzaAX2fNpfK+QX2\nMZ2rOQERTKswNbH0fgylzHDSG14x/Qbf3lmwBb/9K9PJF9jXBM6ct7vW5mTXcpA5mf/6TMmdysnx\nJefsM1LM6xl0ncn5KkdzzAPLoEGYeW/7tjJ9TaUF9/PHTRBsc4u5OnP1NCfGipx4SrPtK9Mvk/91\n2TIT6ofA6M/N/2DLDDPKC6DNMPM7ZJgJ7DlDH1sMyHvO0OHmdYhambctZxikh4+pl1dgXqeq1maI\n60+Pmpb/4xFQu5EZ5aK16azf9pWZUV6nifmMt7rJvM9yrgBL0+J68z/Yna9vKXqziTcBvU1cuu4Z\nc+X0v2B4u6XpaM25EqtEZVp+QGu9BFhSaNsn+f5+E3jTukUrnm9tV27v1oyOzeqVvrNnY3P5nXOm\nrkhaBkxKI+lUyS3N0pTWKq5uHJ3NTOCDv5nAF7PFjCbwbZW3T5OO5vJ143TTAq1V36QrGoTmDe8s\nC9+WJpVUWGaaSbt5Ny96n7ObCQ4HluXlTbfNMem6wL5lO27vv5lRVkueNcP1ej1hjrnrWxMsA3ub\nAH7+qCnDmX1mxFHov0y9f3wYdswtejKJP2CWeMhMMQGpUTszAqNxB3P/iU3mqiion+lwbzUYdswz\nueo+fzf7KGWC/xFL3j2nlZn/bzCpMzD71m1qAvzcUWbI6I2vmv6RnP2zsyv+Po1aaYakos0or/D7\nICbS5KdvfsecmFsNNusWeTc3AT9nwlybm81rHWkZNeYVmPe8rrXNY/N/F8P54yaA5pS/Ydu8tMxf\nC01KsMdjcON/THqn/2T45e/mpJJwwPRp9CxmdFhZOTiak87W2eZKxK2OKZ9yzGvk9HrS/L8dnEz5\nGrUrOtS4ElS7aNPWry6vj2xPLdcynJccHEzLMCbC3K5IWgZMB8qEJaXvV5O0GgyXE8yHN3qLmaVb\nODgM/D/TGb3mdfMhOHfEDHm0Rl+Ck6tJk5V0+dx6iLm8P7XTHPfYn9D57vIFsK4TTe505X/Mh3T/\nL2biW+e7TfAFE2AhLyXTfIA5efn3NGmjSwl5z5eVaSadObubL3pp3MGMOvp+Yl7H8dG1Jhj49zS3\nO99jctU6y4y+yhF0HVw6Y8qVlWmG2H7YueAVwdG14OGbl7ZoEGJGarW4Hpb+wwT67yaYlRVfrQ8L\n7i19JnNJLkSbejQIMa/B0n+Y2cYRM03atP04s991/zCvYexW01rPUaeJeQ+h81Iy+QX2Nu+19MuW\n4x3P61sDEzzPHTblX/Ksea6cwA6WyYCtTDpv8wzLnI62V1fXHG1Hmvk1OVewx9abk1BOv5qTq0nz\njfjEDDNu3u/qG4jlUO2Ce7nV8zffygQVb7l7eOe1MITRcpAJQru+Na3W/CmZHF6BJkBun2tGSwT0\nMUMfr4XgmwBlPnjb54FyMJ205aGUuWpwqQU/PGROUHX9Iai/SdF5Ns5LjRxeZUYL1WuWN6QzPRlm\n35I34Wj9VDMS5uZ3zEiqsXPM6KmEg6aVD+b5/MLzAkTLQSalUKuBGRGWI+fkcuh3+O7evJPndsvz\naG1OPEHXFTwBenjD7fPhptdNGiF2m0k7drrbPNdH3WDZC2a8dlllppkyZGWYzsjRs0zK4tu7zeS3\njreb1AmYyXgtLYvvhdxS8HnaWG4XN0osoI+ZVRwTYeqW03LP0TDMrCE1b6wpz/BPCjYiHJ1MOvDs\nIcucjifKXr+SNO1mYsvuReakE7s1b4CHDVXPVSHLI+es7uRuZiEK63KvZ1qX2+YAumCfR37XPWdJ\nK5w3eeyrnW9QXrXrm6Gf+38xk5NaDrq6K7jaDeCWd00LF6D/83mt/6DrTCoiI9W02vKnYBq1hTsX\nmk7Mz2+Cm141Y6fDRpoWX442t5gT4+rXzdXQyW3Q99m8+x2dYPg007Gb/6rDK8C8x1dMMUFtyFsm\nyOSsL3TuiAliOZ2v+SkFPR81Q07z/z/6/dOchDdNN7nnQS+V7TVa8bIJbGO/ymsEjf0SZlk63rtO\nLLj/0P+ZoYeN2hfc3mWCOUHkLP6Xn38Pc4I+vt4E8oxLBVM3Oa3ws4dg8JvFN8ba3GxOEulJ1hlm\n7OBgUjNbZpgJT9kZVSK414yWO1RsApO4stZDzBtaORQcippfLR8z8uXGV02r7VqXL+4vk54pPKKm\nPMJGmKCsHAu2/oOuM6mprbNNDr1FoXRC835mSF3GJbMwmruXabXnp5Q56SXHwXf3mUBdOCC3HARt\nhlJEy0HmtR85E7o/ZFqjF06YTryjf+SVoSSFPxd1GsNtH5ngumtB2WYZXz5nRgd1vKvgRDq/LuZL\nbwa+WHSUmHdz6P1k0eO71zNj3gt3kIPJaTdqb06iucMg87Xc6wWaTv6APqYvoaT63rXQzIWwVkxo\nO9J8BlZMMf+L4q5gr7EaENwt//iKjJQRV9baMsuzYVjeZXdxQoZZ5zK4vHJmoXr4mlZxRQz/GB7+\ns+Aon5wgvPZ/JkVVXKvNr7OZHxDYF0Z+WnzO1b+Hye2f2GCG3TYtw9r7YE4KT2w1ozzA/D+8m5uR\nREf/MOPkc1ZJLY/248zM7rJ8mfz2uSb92fPRoveF3AJ9nyn/8UsS2MekZRIOmtv50zIODjBxJdwx\n/8r9Ks7uxc+tuFp+XUxD8twR04dS3AKE11gNCO45LXcJ7pXGu7m5vA2tosM8G4SYL2fp+WjZh7iV\nxNnNnMTyq+dvUgOXE0z+taQTnG8wTPjlyqmAQflafs5uZSuTa+2CqQkHR7Osxcltltms111dC7X1\nUPMdBbtK+Rau7GzTavfvVfS1qQwBvU0HZs7SFvlb7mBSMVdqZFQGpfIW18uZLW9j9h/cc87qFR0p\nI67s7h/Mui5VkVJw/1Lrth4Ly2m9F07JlFf91jDqcxPkK6LDHeDubVIFxeXby8LFA0JuNTNu839n\nQGFRK8w4+m4TS97HmgJ6Asoc18On9Nne10r7cWbodUWvDq3E/oN77YamY6rdGFuXRNiznJEfxXUC\nllfbkSX3XZSVi4fJvzs4XX1wBzOBLO0iHMg3ESwjtWAePmKm+Zy1GVb08ZXB3csMedTZRVvtttQw\nDCYfN2v+VAH2H9yVMuOsc8b4ClEZQm6FxyLM+Oaqou+zZhp9Ra5aA/uaoZ67FpjbB5bBuyEwvYeZ\nvHbuqFmUrMuEiqe8ylUuS+rDqwoFd7BuHr+C7D+4C3EtKAX1W5W+37Xk6FTxeRkOjuaqN2o5/PKU\nWVSrThMzKe3rsWaWq3Iwwf1ayslrV6WWexUjwV0IcWXtx5lgHjnLDC+cuNJ8e9nQt01qpP3Ya9+n\nFdjHLHVR0fSVHVOVtL5XqcLDw3VkZKRNji2EKKc/3zWpzZxhrzlyFlOz5rLUZZWVWXA9/hpCKbVV\nax1e2n4175URQpRf36eL326LoJ6jBgb28pC0jBBC2CEJ7kIIYYckuAshhB2S4C6EEHZIgrsQQtgh\nCe5CCGGHJLgLIYQdkuAuhBB2yGYzVJVS8cDxq3y4L5BQ6l72pybWuybWGWpmvWtinaH89Q7QWtcv\nbSebBfeKUEpFlmX6rb2pifWuiXWGmlnvmlhnqLx6S1pGCCHskAR3IYSwQ9U1uM+wdQFspCbWuybW\nGWpmvWtinaGS6l0tc+5CCCGurLq23IUQQlxBtQvuSqnBSqkDSqkopdRkW5enMiilmimlViul9iql\n9iil/mbZ7q2UWq6UOmT57WXrslqbUspRKbVdKfWL5XZNqHM9pdRCpdR+pdQ+pVTPGlLvpyzv791K\nqW+UUm72Vm+l1Cyl1Bml1O5820qso1LqeUtsO6CUuqkix65WwV0p5QhMA4YAocDtSqlQ25aqUmQC\nz2itQ4EewGOWek4GVmqtg4GVltv25m/Avny3a0Kd3weWaa3bAB0w9bfreiul/IAngXCtdVvAERiP\n/VRYh3wAAAKYSURBVNV7NjC40LZi62j5jI8HwiyPmW6JeVelWgV3oBsQpbU+orVOB+YDt9m4TFan\ntT6ltd5m+TsJ82H3w9T1S8tuXwLDbVPCyqGUagrcDHyWb7O917kucB3wOYDWOl1rfQE7r7eFE+Cu\nlHICPICT2Fm9tdZrgXOFNpdUx9uA+VrrNK31USAKE/OuSnUL7n5AdL7bMZZtdkspFQh0AjYDDbXW\npyx3xQENbVSsyjIV+AeQnW+bvdc5CIgHvrCkoz5TStXCzuuttY4F3gZOAKeARK3179h5vS1KqqNV\n41t1C+41ilKqNvA98Het9cX892kzzMluhjoppW4Bzmitt5a0j73V2cIJ6Ax8rLXuBFyiUCrCHutt\nyTPfhjm5NQFqKaXuyr+PPda7sMqsY3UL7rFAs3y3m1q22R2llDMmsM/TWi+ybD6tlGpsub8xcMZW\n5asEvYFblVLHMOm265VSc7HvOoNpncVorTdbbi/EBHt7r/cg4KjWOl5rnQEsAnph//WGkuto1fhW\n3YJ7BBCslApSSrlgOh8W27hMVqeUUpgc7D6t9bv57loM3Gv5+17gp2tdtsqitX5ea91Uax2I+b+u\n0lrfhR3XGUBrHQdEK6VaWzYNBPZi5/XGpGN6KKU8LO/3gZi+JXuvN5Rcx8XAeKWUq1IqCAgGtlz1\nUbTW1eoHGAocBA4D/7J1eSqpjn0wl2q7gB2Wn6GAD6Z3/RCwAvC2dVkrqf79gV8sf9t9nYGOQKTl\n//0j4FVD6v0ysB/YDXwFuNpbvYFvMH0KGZirtAeuVEfgX5bYdgAYUpFjywxVIYSwQ9UtLSOEEKIM\nJLgLIYQdkuAuhBB2SIK7EELYIQnuQghhhyS4CyGEHZLgLoQQdkiCuxBC2KH/Bzzpgzsdl+Q0AAAA\nAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x20adbcf77f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['training', 'validation'], loc = 'upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1792/2133 [========================>.....] - ETA: 0sTest accuracy:  0.341303328715\n"
     ]
    }
   ],
   "source": [
    "cnn_static_best_model = cnn_static(embed_dim=embed_dim , num_tokens=num_tokens, max_len=X_train.shape[1])\n",
    "cnn_static_best_model.load_weights('best_model.hdf5')\n",
    "cnn_static_best_model.compile(optimizer = 'Adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])\n",
    "results = cnn_static_best_model.evaluate(X_test, y_test)\n",
    "print('Test accuracy: ', results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CNN-non-static\n",
    "- Same as above, but the pretrained vectors are fine-tuned for each task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def cnn_non_static(embed_dim, num_tokens, max_len, windows = [3, 4, 5], filters = 100, p = 0.5, l2 = 3):\n",
    "    inputs = Input(shape = (max_len, ))\n",
    "    embedd_intializer = Embedding(input_dim = len(word_idx) + 1, output_dim = embed_dim, \\\n",
    "                     input_length = max_len, weights = [embedding_matrix], trainable = True)\n",
    "    embedded = embedd_intializer(inputs)\n",
    "    \n",
    "    conv_results = []\n",
    "    for window in windows:\n",
    "        x = Conv1D(filters, window, activation = 'relu')(embedded)\n",
    "        x = Dropout(p)(x)\n",
    "        x = MaxPooling1D(max_len - window + 1)(x)\n",
    "        conv_results.append(x)\n",
    "    conv_result = concatenate(conv_results)\n",
    "    x = GlobalMaxPooling1D()(conv_result)\n",
    "    x = Dense(50)(x)\n",
    "    outputs = Dense(5, activation = 'softmax')(x)\n",
    "    m = Model(inputs = inputs, outputs = outputs)\n",
    "    m.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['acc'])\n",
    "    return m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cnn_non_static = cnn_non_static(embed_dim, num_tokens, X_train.shape[1])\n",
    "callbacks = [ModelCheckpoint(filepath = 'best_model.hdf5', monitor='val_acc', verbose=1, save_best_only = True, mode='max')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "history = cnn_non_static.fit(X_train, y_train, epochs = 100, callbacks = callbacks, validation_split = 0.1, batch_size = 50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XecVOW9+PHPd2a2N7ZQdymrIFXqil3xWgJqJBoVMCbi\nVYlGY0xyk2jyS7i5qfcm8RpjwRLjNRZCEEsMthgsxAYoojTpsNTtbXanPr8/ntnCsmXYnWXZOd/3\n68WLnXPOznme2Znvec73KSPGGJRSSsUXV28XQCmlVOxpcFdKqTikwV0ppeKQBnellIpDGtyVUioO\naXBXSqk4pMFdKaXikAZ3pZSKQxrclVIqDnl668R5eXlmxIgRvXV6pZTqk9asWVNqjOnf2XG9FtxH\njBjB6tWre+v0SinVJ4nIrmiO07SMUkrFIQ3uSikVhzS4K6VUHOq1nHtbAoEAxcXFNDQ09HZR4kZy\ncjIFBQUkJCT0dlGUUsdQp8FdRB4DLgUOGWMmtLFfgN8DFwNeYL4x5qOuFKa4uJiMjAxGjBiBfVrV\nHcYYysrKKC4uprCwsLeLo5Q6hqJJyzwOzOxg/yxgVOTfAuDBrhamoaGB3NxcDewxIiLk5ubqnZBS\nDtRpcDfGvA2Ud3DIbOAJY70P9BORwV0tkAb22NLXUylnikXOPR/Y0+JxcWTb/hg8d58RDhtCxhAK\nG8KRry4UbGANhcMEI/tdCC6X4BJIdLtI8LhwtQjAxhiCYUMwFCYQNrgAj9uFxy24RbodrENhQ2mt\nj7QkD2mJbkSEQChMhdeP1xeiIDsFj7v9a35DIITbJSR0cMyxFAyFKa31U+sLUOsLUecLUukNUO71\nU1nnx+US0hLdpCZ5wIAvGMIXDBMKt/31kkkeF9lpieSkJdI/I4mC7FTSk2LbNWWMwRcM4w+FSU/0\n4HK1/TdtCITYeqiWsDEMzU6lX2pCj16sw2HDwZoGiivq2VPuZX9VA7lpiZw4IJ0T+6eTlZKAS2yD\noao+wJ5yL8UV9fiC9j3hFiErNYGh2akMykqO+j0SCht2lNays9SLCLhdgksErz+E1x+kPhAiweUi\nKcFFottFIGzwBezfseVfcUBGEoV5aQzLSSU5wd20vdLrZ2eZl11lddT6gnhcgtvlitTFHuMSIcnj\nJinBfh79wTANgRD+yHslGDaIQHZqAtmpiQzOSmFYbmoMX/3YO6YdqiKyAJu6YdiwYcfy1FGprKzk\n6aef5hvf+MZh28PGNAXgsDHU+YLUNASp94cIhg03XvNlfvWHR8jMymr3ue//7S+ZduoZnHb2jMO2\nC0KCR8DYN3mog++0TUl0MzgrJapgU9sQpLohAEBVfYC7ln3Khv3VbD5QTUMgDNgPUUqCm1pfsOn3\nUhPdTCrox8ShWSR73ITCBl8wxI7SOjYdqKG4oh6AzGQPOWmJnDIih6tPGUrR8OymwBMKm6Yg0Fp1\nQ4Di8nqKK7zsLveys6yOXWVeUhPd/OqKieSkJbZfJ1+QVTvLeX9bGat3VbC3op5DNQ20E6djpl9q\nAjmpiXj9Ier8QdISPVx3xgi+evrww/4WDQH7Om0rqWVnaR2ltX7K6/xUeCP/1/kp9/qbXn+wF/j8\n7BQKslNI8ribGgJ7K+rZWVZ3WN3SEt0M6ZdC/4wk+mckMSDyf/+MJDwuF8UV9nUtq/Xjdgsel5Dk\ncZGXbo9JT/JQUuvjQFUDh6p9BEL2XL5giANVDeytrCcQ6vzFdLuk3YtjI5fAif3TOXNkHmePymP0\noAwaAiFqfSHKan1sK6ll66FaPj9Yy6YW78lYSYxcWAwmqjp1xcSCLOZNH8bFJw9mf1U96/ZUselA\nDYFQc10S3PailOxxk5XiaWo4FOalUZDdsxcHieYLskVkBPBSOx2qDwFvGmOeiTzeDMwwxnTYci8q\nKjKtZ6hu3LiRsWPHRl34WNu5cyeXXnopn332GQCBUJjd5V6q6hpITEhoelOHjUFESE1wkxBpVdvW\ngDS1OsC+sTD2w+B2u3CLYDCEwxAyBn8wbP+Fwgg0/b7HJba17pKmlrw/FKas1k8gFCYzOYHstAQE\nQSTyBvK4EBHCxnCwuoGSGh8usfsP7NrGd18vZcygDMYNzqIwLxWvP0R1QwCvP0S/lERy0hJI8rhZ\nv6+Kj3ZXsmF/NaGwaSrPsJxURg/K4KSBGYSNoaLOz8FqH+9sKaHOH6IwL42BmUkUV9Szv6qBQZnJ\nzDllKFcVFeBxuXj+470sXVPM5oM1h73mWSkJjMhNZdOBGgrz0njyxlPJS0867JgDVQ384Z9bWLJ6\nD4GQIcEtTCrox4i8NAZnJTMwM5mslATSktykJnrITk0kOy2BfimJGAxeX4haXxBXJNgleuxr25Z6\nf4gKb4AKr78p4O0p91JZH7B3AIketpXU8s6WUvqlJjB70hAOVvvYfLCGXa2CcUbkApidmkhuWiLZ\naYlkpyaQmughKcFFgstFaZ0vEpTrCQTDeNz2/TMoM5nRgzIYPSgDt0uaWtMHqxs4VOPjUI39G7cO\nijlp9lzhyF1kfSBEWa2fYIuCZSZ7GJCZTFLkdUhwuxiUlUxBdioF2SkMzbH/D8lKobTWx9ZDtWwr\nqcUbacwEQ2H6RVroBdmppCXZRkAgZKj0+m1ZK7x8UlzFhzvK2g3ceelJjByQxvghWYwbnMkJ/dNw\niRAMG4wxpCS6SU/ykJzgJhAK27udYLjp/d74ngd7N7S/qqGpsVAfCDW/JqmJDM9NpTAvjcyUBNuI\nivxrFAzbz6MvGCJsjG3FN75XIp9F+74PUF7nZ/PBGpas2nPE+zklwU1KorupTIGQvXi2vsB8/dwT\nuGtW12KdiKwxxhR1elwMgvslwG3Y0TKnAvcaY6Z39pzHY3CfO3cuL7zwAqNHj8bt8YA7kcysLHZv\n38rKNeuYP+9q9u8rJuD38a3bv8XNN38daF5Koba2llmzZnHWWWfx7rvvkp+fzwsvvEBKSgrz58/n\n0ksv5corr2TEiBFcd911/O1vfyMQCPDXv/6VMWPGUFJSwjXXXMO+ffs4/fTTef3111mzZg15eXmA\nvW0urfNRUu07ooWf4HaRkeyh3h+iPhAiN83eOrpc0qXXNRy5De0sDVDnC7L80/08v3YvvkDYBoV+\nKawrrmLl1lJcQtMHdsqwflw0bhDDcyNBJDuV7EhLfeWWUm58YhVDs1N56qZTSUlws+lADa+tP8AT\n7+0ibAxXFw1l1oTBTBue3fQB6i0f767gvn9u5c3PS+yFb2AGJw3KYOSAdEb2T6cwL63Hy2iModYX\npKTGRyBkyM9u+64uHDZU1georg/QPyOJtBinmTrSEAjx0a4K9lR4SU30kJbkJislkZH908lK7dvD\nc40xfLynkrc2l1CYl8bJBVkU5qa1mWYLhsJU1Qcid3H271CYl9al88YsuIvIM8AMIA84CCwEEgCM\nMYsiQyHvw46o8QLXG2M6XTSms+D+07+tZ8O+6s6e5qiMG5LJwi+Ob3d/Y8t95Ycfsezvr3HbdXNY\n8/EnjB09EoDy8nJycnKor6/nlFNO4a233iI3N/ew4D5y5EhWr17N5MmTufrqq7nsssu49tprjwju\n3/3ud/nmN7/JAw88wEcffcSjjz7KbbfdRn5+PnfddRevvPIKs2bNoqSkpCm4NwqGwwQi+UYTySPX\nNASpbQgiIhRkp5CZ0vzB6a2L5u4yL0vX7CEYNlwxNZ+RAzI6PP7dbaXc8PhqRMDrty0vl8DlUwq4\n44JRDM05/nKc4bBpN2euVE+INrh3egk3xszrZL8Bbj2Ksh3Xwgb2VNST5HEzffr0psAOcO+99/Lc\nc88BsGfPHrZs2UJubu5hv19YWMjkyZMBmDZtGjt37mzzPFdccUXTMcuWLQNg5cqVTc8/c+ZMsrOz\n2/xdj8uFJ7G5syotyUNOWpJNF3H8jJAZlpvKdy4aHfXxZ5yYx5M3TueZD/cwIjeVcUMymZCfxYCM\n5B4sZfdoYFfHq+NqhmpLHbWwe0ogFCYQCpPgFgZlJpGe3nzb9Oabb/KPf/yD9957j9TUVGbMmNHm\n+PGkpOZ8sdvtpr6+vs1zNR7ndrsJBoNtHnO0XMdJUO+OacNzmDY8p7eLoVSfd3yMZzsOhI2hMuCm\nrraGEblpuFsN46qqqiI7O5vU1FQ2bdrE+++/H/MynHnmmSxZsgSA1157jYqKipifQynlDMdty/1Y\n21/VQFJ6FmeecSZFUyaRkpLCwIEDm/bPnDmTRYsWMXbsWEaPHs1pp50W8zIsXLiQefPm8ec//5nT\nTz+dQYMGkZHRcZ5aKaXaEtVomZ5wPI2WCYbCbDxQQ3ZqQo+PPe2Iz+fD7Xbj8Xh47733uOWWW1i7\ndm23n7e3RyEppWInZh2qTlDhDWCMIbfV+Opjbffu3Vx99dWEw2ESExN55JFHerU8Sqm+y/HB3RhD\nhddPaqKblITeHTs9atQoPv74414tg1IqPji+Q7XeH6IhECI7tf1p70op1dc4PriXe/24ROjXx2fL\nKaVUS44O7qGwocobICslAbfL0S+FUirOODqiVdUHCBnT4UqESinVFzk6uFd4/SR53KR2cYGn9PR0\nAPbt28eVV17Z5jEzZsyg9ZDP1u655x68Xm/T44svvpjKysoulUkppcDBwT0ctkvBZqV4ur0Wy5Ah\nQ1i6dGmXf791cF++fDn9+vXrVpmUUs7m2OBeHwhhMKQkNo8GvfPOO7n//vubHv/nf/4nP//5zzn/\n/POZOnUqJ598Mi+88MIRz7Vz504mTLCrIdfX1zN37lzGjh3L5ZdfftjaMrfccgtFRUWMHz+ehQsX\nAnYxsn379nHeeedx3nnnAXYJ4dLSUgDuvvtuJkyYwIQJE7jnnnuazjd27Fhuuukmxo8fz0UXXdTu\nGjZKKWc6fse5v3wnHPg0ts856GSY9WugeUnZlimZOXPmcMcdd3DrrXaRyyVLlvDqq69y++23k5mZ\nSWlpKaeddhqXXXZZu639Bx98kNTUVDZu3Mi6deuYOnVq075f/OIX5OTkEAqFOP/881m3bh233347\nd999NytWrDhiad81a9bwpz/9iQ8++ABjDKeeeirnnnsu2dnZbNmyhWeeeYZHHnmEq6++mmeffZZr\nr702pi+XUqrvcm7L3R8iwe067Hsep0yZwqFDh9i3bx+ffPIJ2dnZDBo0iB/+8IdMnDiRCy64gL17\n93Lw4MF2n/ftt99uCrITJ05k4sSJTfuWLFnC1KlTmTJlCuvXr2fDhg0dlnHlypVcfvnlpKWlkZ6e\nzhVXXME777wDRL+0sFLKmY7flnukhd1TvIFgmx2pV111FUuXLuXAgQPMmTOHp556ipKSEtasWUNC\nQgIjRoxoc6nfzuzYsYPf/va3rFq1iuzsbObPn9+l52kU7dLCSilncmTLPRiy38XY1tegzZkzh8WL\nF7N06VKuuuoqqqqqGDBgAAkJCaxYsYJdu3Z1+NznnHMOTz/9NACfffYZ69atA6C6upq0tDSysrI4\nePAgL7/8ctPvZGRkUFNTc8RznX322Tz//PN4vV7q6up47rnnOPvss7tTdaWUQxy/Lfce1Pjlualt\nrCUzfvx4ampqyM/PZ/DgwXzlK1/hi1/8IieffDJFRUWMGTOmw+e+5ZZbuP766xk7dixjx45l2rRp\nAEyaNIkpU6YwZswYhg4dyplnntn0OwsWLGDmzJkMGTKEFStWNG2fOnUq8+fPZ/p0+5W0N954I1Om\nTNEUjFKqU45c8vdgdQMHqxsYPyTTETNTdclfpeJHtEv+RhXZRGSmiGwWka0icmcb+7NF5DkRWSci\nH4rIhK4U+lip94dI8rgdEdiVUs7UaXQTETdwPzALGAfME5FxrQ77IbDWGDMR+Brw+1gXNFaMMXj9\noS7PSlVKqb4gmqbrdGCrMWa7McYPLAZmtzpmHPBPAGPMJmCEiAykC3o6TRQIhQmGw44J7r2VdlNK\n9a5ogns+sKfF4+LItpY+Aa4AEJHpwHCg4GgLk5ycTFlZWY8GpMbJS22NlIk3xhjKyspITk7u7aIo\npY6xWI2W+TXwexFZC3wKfAyEWh8kIguABQDDhg074kkKCgooLi6mpKQkRsU6UlV9gFpfEE91crfX\nlOkLkpOTKSg46uusUqqPiya47wWGtnhcENnWxBhTDVwPIDZi7gC2t34iY8zDwMNgR8u03p+QkEBh\nYWG0Ze+Sqx96D38wzPO3Tu38YKWU6qOiScusAkaJSKGIJAJzgRdbHiAi/SL7AG4E3o4E/OOKMYYN\n+6o5OT+rt4uilFI9qtOWuzEmKCK3Aa8CbuAxY8x6Ebk5sn8RMBb4PxExwHrghh4sc5d5/SFqfUHy\ns1N6uyhKKdWjosq5G2OWA8tbbVvU4uf3gJNiW7TYK6v1A5Cr37yklIpzjprFU1LrAyAvI6mTI5VS\nqm9zVHAvawzuaRrclVLxzVHBvbQxLZOuaRmlVHxzVHBvbLlrcFdKxTtHBffSWh8ZyR6SPPE/O1Up\n5WzOCu51fvqna75dKRX/nBXca3yaklFKOYKjgntZnZ88bbkrpRzAWcG9VlvuSilncExwD4TCVHgD\n2nJXSjmCY4J7RV3jGHcN7kqp+OeY4N649EB/TcsopRzAMcG9adEwbbkrpRzAMcG9tHF2qq4IqZRy\nAMcE98aWu64IqZRyAscE99JaH4luFxlJsfraWKWUOn45KLj7yUtPdMSXYiullIOCu087U5VSjuGY\n4F5W5yNPh0EqpRwiquAuIjNFZLOIbBWRO9vYnyUifxORT0RkvYhcH/uidk9ZrV9b7kopx+g0uIuI\nG7gfmAWMA+aJyLhWh90KbDDGTAJmAL8TkeOmmWyMoaxWFw1TSjlHNC336cBWY8x2Y4wfWAzMbnWM\nATLE9lamA+VAMKYl7YbqhiD+UFjTMkopx4gmuOcDe1o8Lo5sa+k+YCywD/gU+JYxJhyTEsZA4wQm\nbbkrpZwiVh2qXwDWAkOAycB9IpLZ+iARWSAiq0VkdUlJSYxO3bky/WJspZTDRBPc9wJDWzwuiGxr\n6XpgmbG2AjuAMa2fyBjzsDGmyBhT1L9//66W+ag1Lz2gLXellDNEE9xXAaNEpDDSSToXeLHVMbuB\n8wFEZCAwGtgey4J2R1ljWiZDW+5KKWfodC6+MSYoIrcBrwJu4DFjzHoRuTmyfxHwM+BxEfkUEOAH\nxpjSHiz3USmp9SMCOaka3JVSzhDVQivGmOXA8lbbFrX4eR9wUWyLFjtltT6yUxPxuB0zZ0sp5XCO\niHZltX5d6lcp5SiOCO6ltT4dBqmUchRHBPeyOr8Og1RKOYojgntpjbbclVLOEvfB3RcMUeMLas5d\nKeUocR/cK+oCAORoWkYp5SDxH9y9dumBbB3jrpRyEMcE936pCb1cEqWUOnbiPrhXem1aRlvuSikn\nifvgrmkZpZQTxX9wr9O0jFLKeeI/uHsDpCa6SU5w93ZRlFLqmHFAcPdrSkYp5ThxH9wrvQFNySil\nHCfug7u23JVSThT/wb3Ory13pZTjxH9w9wbI0XVllFIOE9fBPRQ2VDcE6KdpGaWUw8R1cK+qD2AM\nZGtaRinlMFEFdxGZKSKbRWSriNzZxv7vicjayL/PRCQkIjmxL+7R0dmpSimn6jS4i4gbuB+YBYwD\n5onIuJbHGGN+Y4yZbIyZDNwFvGWMKe+JAh+NSl00TCnlUNG03KcDW40x240xfmAxMLuD4+cBz8Si\ncN1VXqeLhimlnCma4J4P7GnxuDiy7QgikgrMBJ7tftG6rzEto6NllFJOE+sO1S8C/2ovJSMiC0Rk\ntYisLikpifGpj6RpGaWUU0UT3PcCQ1s8Lohsa8tcOkjJGGMeNsYUGWOK+vfvH30pu6jCG8DjEtKT\nPD1+LqWUOp5EE9xXAaNEpFBEErEB/MXWB4lIFnAu8EJsi9h1lV4//VITEZHeLopSSh1TnTZpjTFB\nEbkNeBVwA48ZY9aLyM2R/Ysih14OvGaMqeux0h6lirqAjnFXSjlSVPkKY8xyYHmrbYtaPX4ceDxW\nBYuFcq+fbO1MVUo5UFzPUK30+rXlrpRypLgO7hXegI5xV0o5UtwGd2NMU4eqUko5TdwG9zp/iEDI\naFpGKeVIcRvcK+p00TCllHPFb3BvXBFSR8sopRwojoN746JhmpZRSjlP3Ab35nVltOWulHKeuA3u\nzTl3bbkrpZwnfoN7JC2TlaLBXSnlPHEb3Cu9frJSEvC447aKSinVrriNfOVeXTRMKeVccRvcdXaq\nUsrJ4ja4V+iiYUopB4vf4F6ni4YppZwrboO7pmWUUk4Wl8HdFwxR5w+Rk6ZpGaWUM8VlcK+MjHHX\nlrtSyqniOrhrzl0p5VRRBXcRmSkim0Vkq4jc2c4xM0RkrYisF5G3YlvMo1PRtK6MpmWUUs7U6Rdk\ni4gbuB+4ECgGVonIi8aYDS2O6Qc8AMw0xuwWkQE9VeBoVGpwV0o5XDQt9+nAVmPMdmOMH1gMzG51\nzDXAMmPMbgBjzKHYFvPoaFpGKeV00QT3fGBPi8fFkW0tnQRki8ibIrJGRL4WqwJ2RUVTh6q23JVS\nztRpWuYonmcacD6QArwnIu8bYz5veZCILAAWAAwbNixGpz5SpddPosdFSoK7x86hlFLHs2ha7nuB\noS0eF0S2tVQMvGqMqTPGlAJvA5NaP5Ex5mFjTJExpqh///5dLXOnKiOLholIj51DKaWOZ9EE91XA\nKBEpFJFEYC7wYqtjXgDOEhGPiKQCpwIbY1vU6FV4/fRL0Xy7Usq5Ok3LGGOCInIb8CrgBh4zxqwX\nkZsj+xcZYzaKyCvAOiAMPGqM+awnC96RSm9A8+1KKUeLKudujFkOLG+1bVGrx78BfhO7onVdZb2f\nE/LSe7sYSinVa+JyhmqFttyVUg4Xd8HdGEOVN6DryiilHC3ugrvXH8IfCusXdSilHC3ugruuK6OU\nUnEY3HW5X6WUiuPgruvKKKWcLO6Cu6ZllFIqDoN7Zb0uGqaUUvEX3OsiLXddfkAp5WBxF9wrvAHS\nEt0keuKuakopFbW4i4CV9X4dKaOUcrz4C+669IBSSsVfcK/w+nUYpFLK8eIuuFdpy10ppeIvuFd4\n/RrclVKOF1fBPRw2VNUHNC2jlHK8uAruNQ1BwkbXlVFKqbgK7k1LD6RoWkYp5WxxGdyz0zS4K6Wc\nLargLiIzRWSziGwVkTvb2D9DRKpEZG3k309iX9TONa8ro2kZpZSzdfoF2SLiBu4HLgSKgVUi8qIx\nZkOrQ98xxlzaA2WMWqWmZZRSCoiu5T4d2GqM2W6M8QOLgdk9W6yuqajTtdyVUgqiC+75wJ4Wj4sj\n21o7Q0TWicjLIjI+JqU7SpX1AUQgU1vuSimH6zQtE6WPgGHGmFoRuRh4HhjV+iARWQAsABg2bFiM\nTt2s0usnMzkBt0ti/txKKdWXRNNy3wsMbfG4ILKtiTGm2hhTG/l5OZAgInmtn8gY87AxpsgYU9S/\nf/9uFLttld4A2To7VSmlogruq4BRIlIoIonAXODFlgeIyCARkcjP0yPPWxbrwnbGLj2g+XallOo0\nLWOMCYrIbcCrgBt4zBizXkRujuxfBFwJ3CIiQaAemGuMMT1Y7jZVegPkpmtwV0qpqHLukVTL8lbb\nFrX4+T7gvtgW7ehV1vsZOSC9t4uhlFK9Lq5mqFbW6XK/SikFcRTcA6EwNb6gfjG2UkoRR8G9KrL0\ngK4ro5RScRTcG5ceyNIJTEopFT/B/UCVD4CBmcm9XBKllOp9cRPc91Z6Acjvl9LLJVFKqd4XR8G9\nAZfAoCxtuSulVPwE94p6BmYmk+COmyoppVSXxU0k3FdZzxBNySilFBBHwX1vZb3m25VSKiIugns4\nbNhfpS13pZRqFBfBvaTWRyBkyM/W4K6UUhAnwb24oh6AAm25K6UUECfBfV+lDe6allFKKSsugvve\npuCuY9yVUgriJbhX1JOZ7CEjWdeVUUopiJPgvq+ynvzs1N4uhlJKHTfiIrjrGHellDpcHAV3zbcr\npVSjqIK7iMwUkc0islVE7uzguFNEJCgiV8auiB2rbghQ0xDUMe5KKdVCp8FdRNzA/cAsYBwwT0TG\ntXPcfwOvxbqQHdlbocMglVKqtWha7tOBrcaY7cYYP7AYmN3Gcd8EngUOxbB8nWoc4645d6WUahZN\ncM8H9rR4XBzZ1kRE8oHLgQdjV7ToNI5x17SMUko1i1WH6j3AD4wx4Y4OEpEFIrJaRFaXlJTE5MR7\nK+tJdLvIS0uKyfMppVQ88ERxzF5gaIvHBZFtLRUBi0UEIA+4WESCxpjnWx5kjHkYeBigqKjIdLXQ\nhxWuop4h/ZJxuSQWT6eUUnEhmuC+ChglIoXYoD4XuKblAcaYwsafReRx4KXWgb2n7NUv6VBKqSN0\nmpYxxgSB24BXgY3AEmPMehG5WURu7ukCdmafTmBSSqkjRNNyxxizHFjeatuido6d3/1iRccfDHOo\nxqedqUop1UqfnqF6oKoBY47hGPdVj8LirxybcymlVDdE1XI/XhVXeoFjNMbdGPjX76FyN5RugbxR\nPX9OpZTqoj7dct9TboP7sJxjsCJk8Sob2AE2v9zz51NKqW7o08F9d7kXj0sYnHUMFg37dCm4kyB3\npAZ3pdRxr08H9/oDW3gn6Vt4Sjf17IlCQVi/DEbPhAlfhj3vQ11Zz55TKaW6oU8H99ySDxhsDsG7\n9/bsiXa8BXUlcPJVMHoWmDBsebV5f8lmeOt/IBzq2XIopVSU+nZwr91if/h0KdQc7PwXAvXw/Dfs\n8Ufj06WQlAkjL4TBkyFjMGyOjAwN+uGv82HFL2Dj347ueZVSqof02eBe3RBgRHgXNUmDIBy0wxQ7\nEg7Bsptg7VOwbAFs/Ud0JwrU26A99jJISAYR23rf+k8INNgRNIc2QHI/WPm/dlRNS60fK6XUMdBn\ng/uesjpGyx4qhpwDJ82E1X+0wbY9r/0/G6TP+38wYBwsuQ72f9LxSYyBjS+BvwZObvH9I6MvhkAd\nrH4M3v4fGH85XPhfsH8tbH+z+bgNL8Avh8Bfr4fd73cv0PvrbO5fKaWi0GfHuR/at5PxUkv94Akw\ncjI88TK8o7LSAAAUS0lEQVR8+leY+tUjD37vAXj/ATj1Fjj3ezDlWnj0Anjqarjkd+BJsoG3rgTK\nt0P5tsj/O8BXDekDofCc5ucrPAcS0+HVH0JyJsz8b0jpByt+aVvvJ55n8/DPfwMyBsHWN2yH7OBJ\nMPt+GHTy4eU78KkdhZPQznj9+gp46Bxbjq+9CIn6ZeBKqY712eBeX/wpAJnDJ9lgO3ACvP+gDdzS\nYoXIyj3w2o9gzKXwhV/YbZmD4St/hcdmwl9azTgVN/QbBrknwtDTIOcE+/wud/MxniQ48d9g44tw\n0S8gY6Ddfvqt8PqPYcfb8Pfv2mB93Us28K/7C7z1G3vOK/8EJ10Evhp4+U5Y+yTkjYYvPwqDJx5Z\n2b//B1TttXV59kaY82dbHl8NvPZjmzq65HeQlB7DV9jhqvZC9V4YOr23S6JUl/TZ4C4lGwBIHzrR\nBvPTboEXboXtK2zgbbT2Kdsq/8IvDw/QA8fB7R9Bxc7mbSnZNrC7EzovwNnfsReUKdc2b5s2H975\nLTx5JYQD8LUXICvyvSZF/w4nzYJn5th/Z30bPltmz1/073bs/KPnw/kL4bRvgCuSMVv3V/hsqU0n\nJWfBy9+Dl38AE+fAshubJ1Yd2gDX/AUyhxztS3l8MgaKV8OG56Fsq72Tqj0EBafAqAth1EWQU9jx\nc/hqbTqrUVr/5te1I0Ef/PlyKP0cvvQgTJ7Xvboo1QvE9FKHX1FRkVm9enWXf3/lb65kbP0acn+y\nw24I+uCeiTBgjA2qYDtRfz/Jpjy+dkxWIIY3fmYD/IX/BWd+68j9/jrb+t68HLKGwuUPwYgzwVsO\nL34TNr0EeSfB9AUw4mz440W2TvOXg9tj+w7e/QMg9ve//Ag0VMPS6+2Inmv+0nbr/3jQUG07v1Nz\n2j8mHIZPl8AHi2Dfx+BJtn+/nELbab3rXzbQI3DlH+28g5YObbT9JFtft7OKW35/zIDxMPOXcMKM\njsv55q/hzV/Z4w9tgC89AJOv6fh3VPeEQ7ahk3PC4Xfe6ggissYYU9TpcX01uH/+X1MIJucw7vtv\nNG9ceQ/8YyHctALyp9oRMU9+2aZBJlwRg1JHIeizAeiE89p/k4ZDtqU+4iybsmlkjM3Nv/sHG9jA\n5vZvXtncSg2H4eXvQ8gHF/3ctubB5u2fngPeMrv9lBtj9yExBg6ujwTM1TBpHoy9tO1jQwEbVN2J\nMGSKvVsKBWzn84pf2vrctqr9foMPH4Hl/9F8gZs078h0U9k2eO5m269xy7+gX+S7ZDa/Aovn2YA+\neDKMvKD5TiZQDx8+ZO90TpoF5/wHDJl6ZEv+0CZYdBaMmw2z74Nn5sL2t2xfyRRdNK5H+Gpg6Q12\n7sjgSTD96/ainXAMZp7HUvV++94de5mdE9PeXeKnS226r9+wLp0mroN7KBgg+LPBfDLkKqZ/vcXX\ntjZUw/9OgBNnwNVP2BExO96G726yefK+ojEl8fETdiTQmEui+73aQ/D8LfaidtJMuOw+SO9/dOdt\nfUFoqIbHvmBbsGBTVw3Vtn+g8YIZDtuL0vrnbCD010SOzbEpsgOfQulmyJ8Ge9fY1NPZ32m7DH++\nHKqK4dYPO744le+wQXjIFNvJvP9jePxSe1G4ZklzP0hLgQb44EF4+3e2jKm5cOL5Ns1z4vm2bn+a\nadMxt66yr12gHp6ZZyeyzXkKxlwc/evZ2w58ausy9rLoUo29oaoYnp5r31/Tb7KjzUo2QWoeXHq3\nvcgea219DjoTDsOTV9i0MNj3+sxfH9lnU3MQ/necvYDN/GWXihdtcO+TOfeSXRsYJAHC/ccdviM5\nE065wY5Y2f0BbPq7bf31pcAO9o019BT772ikD4Br/gofPgyv/wTuHmNbBzknQv/RdhTP8LNsi6i2\nBLb9E3attC3h8u12VM4Nr9nWU6PNy+0H78Kf2dZIUgY8daVNLYGd0PXqXfZOI7MATv6ybTEHffYi\ns/UNe3cyb7G94Dwzz95hTZt/ZHom0AC73rX7Ovtw5RTCrP+2/Syv/ciOlErLaz+wg633Wd+GqdfZ\nsm15Hba9YdNAiE3/lG2xefbGi2JCCsx9yl44lv47XP93+8HtTH3l4XdlPWnvR/bi2m+4HQjQUGXv\ngHb9y+7PnwZXPGL3HU/2rIK/XGtTlV9ZYt83xtgG2T8WwpKv2T6tmf9t796Cflu3jhos4TDU7IPE\nNHuxBrtUyEePw+rH7UXulBtg8leO/Pv4vfDCN6B4Dcx98vDPQWc+fMgG9kt+Bwmp8I+fwh8vhKse\nt0OlG330hE1NFv179M/dRX2y5b75n08w+u1vsnbWC0w+dcbhO2sP2dZ7UrpNUXzjfRgwtvsF7msO\nbbQBrzFwl2y2qRxPCmSPsK0jjP0A9B9jc52fLbO55Uvvbn6ep+fYlMwdnzYHXF8tPHUV7H7PPkfG\nELjwpzDhyiNvRVu3gg5thAfPsJ3GjaOXGm1bAX/+kg3QJ32h8zoaA0u+aucvJPeDG16H/icd3esU\nDttW/5Z/2ICfPdwGwtYXl9pDdvhswGvP015nrq8WXrkTPv6zvXNqOTS3oQpeXwgmZC+42cPtRaB8\nu8039xtmA9zwM+x5tq2wASMz347ESspo+5yPzYz8LVroN9y2hNMGRNJ4AbhgYXOAT8mxqctjYfMr\n8Pkr9rXIn2bna6y82/ZtZOXDvL/YAQ4thQK23+Odu20Dwp0AVXtsyu20W+17p/FvVF8Jr/4I9q62\nd3QhX3Mds4fDwQ12W+G5EPLb1yohDSbNtY2/AWNsi/qZubaRkpZnA/2Vf7QTFjtzcAM8PMM2nuYt\ntuXy1do73kC9vQt1e2y9fz/JLhfejT7AuE7LfPbUDxj7+UPsv3UbBQNyjzzgpW/bHG/BKXBjlDNR\n453fCztX2rx56RYbQEZeYHPTjQH52Rtta/Y/Prd3O/WV8NtR9gPQOhD7au1wz5wT4IzbbEspWi/c\nCuuWwDfXHJ53fP0ndk7CD3ZGP6zTW27LcerNMOzU6MvQFaVbbGssJQeuX27nMLS0dw08e5MN1tkj\noHof3PCqTR0F/fD0VbDjHXtB9ZY2/5470XaOV+2xwceTYoORCdtO8sa5Fv/2Y3vxbTnqq+RzuP8U\nuOCnMPFqe+5w0HbGNx5XVWz7KHa+c3h5L38YJs2J3etTshnEdfh3HQT9cO9kO6wUIL/IHlP8oW0M\nXPK7ju9wdv4L3okck3OircsnT9s5KzN/BZW77HyV8u22QZBzgn3t/XWRi+YOe0d2yk02iIOdvPjB\nQzb3HfLZDvaybbYx+OU/2oveM3Nh31o49ev2b1K+3dblrDvsSK3GC0tdKTwxG2oPwi3vHX5XsfFv\n9s7k8ofshWTT32HxNTDnSRj7xS6/zHEd3D+/dzbuss8Z/pP1eNxtdFqU74AHz4TL7j18Zqnq2NY3\nbN7w6idsrnPtM/D8zXDjG1DQ6XspelXF8IdpMO5LcMVDzdsXnW1bp9cvb/93e9vuD2y/QOYQmP+S\nDfDhEPzrHtthnD4QrnjY3g09dK69cC54y054++QZmP2A7ZhtqIKKXTZoZebbQOyvs8F/2z/t6zDq\nQhsM96+1dwPFq+zs6LlPNweX1/6fnd/xnY02LdeecAgOrLMtYrDDaav3wTdXH35HEArasjQ+fzhs\nh+K++St70cg5wQbZydcc/p7Y/wn86WJ7kb9ttU2RAqx92vYDXfmYTQV++LANiBf/pmsXFmPsa/n+\nAzZNuG2FLdecJ6Hw7KN7rrpSWPM4rPojYGyre8hku8/vhee+bueypGTbenvL7cXixH+zDZ4NL8Jn\nz9phz3OfsavGthQO28mHAa9tvT99le2wv+NT25LvorgO7od+MY714eGc9+MO1lUP+vperr23hUPw\nv+Nta/6axW2nZGLl9Z/Av+6Fm9+xM3brSuE3J9rx/Od+L7bnirVd79q5DJmDbavs9YW272Lcl+CL\n9zTneovX2A7a1Fyo2Q8zfggzftC1cxoDb//GLlA35yk7Winot51zQ0+1/QJHo3i1nVdx5h02pQa2\nZfnsTZCWaxfJKzjFrtm0d7XNP+eOsi3Y0i22xXvZfTZAV+6xKSsThrpDcMbtcNHPbHB74DSbUrl5\npX0PhSNDU6OZb9DRa9EY4LML7YTE7nwzWjhkLxBtxQt/XfNdadBvlzl581f24pyQZudATF9g+7Ta\nsvElO1HynO/bpUrO+xGc+/2ul5UYd6iKyEzg94AbeNQY8+tW+2cDPwPCQBC4wxiz8qhLHQ1/HXmB\nfZRnnt/xcRrYj57LbW/t37sfSrfalvypX++ZccdnfRvW/J/teLp2afOaPCeeF/tzxdrwM+DaZ23H\n8qPn2+GdX3rQDtts+VoVTINZ/wMv3WE7BrvzoRaBs75j+0VevQtGnm9TaHUltoP4aBUUwaRrbICc\n+jXbB7PkOts/lVUAnyy2gSx9kK3bxLnNAdlbbjs7n1tgf2/zcptbvuFVePc+eycx9To7Qqp0s011\nNL4u3QnqLV+LL0TmKxSc0vG8iWi43IenulpqmW70JNrJkhPn2BTnCec2D0Vuz5hLbOPl7f8Bl8e+\n1sdIp8FdRNzA/cCFQDGwSkReNMZsaHHYG8CLxhgjIhOBJcCYnigwJZtwYajPaedKqbpn0jy70uWy\nm+zt5vgemh+Qkg1nf7d5uYbtK+wHZciUnjlfrA0/Hb76HKz+k73TyDmh7eOmzYdhp9khmt29SLo9\ndoTQE5fZILrnA9uZPbKThk57Llho88JLvmbz5YMn2jolZ9k73wOf2RZp6/6P1By4dhn87Vu2Y9SV\nAF9dZi8MFyy0qYxX7oSGStuxO+5L3at3W0Si63TvCak5MO6y6I4VgXPvtK33MZce2U/Tg6JpuU8H\nthpjtgOIyGJgNtAU3I0xtS2OTwN6LNfjPfA5qYBr4PieOoWzDRhr0zL7PoKsYT07omL6Atux9fpC\nOxql9Ro+x7uh0ztfe0YktqO1TjjX9oe88zsINsA53+v6a5YxyN5NvP5jm9v/6rLmlqgnyd55tMeT\naGfuDjvVDoFtXFgvfQCc+wM7PBXg4t92K78cF8ZcYju8u9GJ2hXR3CPlA3taPC6ObDuMiFwuIpuA\nvwNtDuIUkQUislpEVpeUlHSlvOwYfDETGx6mX/5RDnlT0ZsUWUtl/OyenQqekAzn/dBeSKqLD18T\nSLXvop9j20+m+7NmT7sFrni0ucV+NETsncmoCw7fPn2BvVNJ63/42ktOJWJH2RzjeQYxW8/dGPOc\nMWYM8CVs/r2tYx42xhQZY4r69z+KmZMt7Cn3Uk06w3J1BcQeM2mOvYU8BhMtmDTXrq8PdskG1bl+\nw2wu/6xv22F/3eFOgIlXNY9uiQVPor1YzF/e/jLWqsdFc7+0Fxja4nFBZFubjDFvi8gJIpJnjClt\n77iuGjMok4VfHMeIvKMYV62OTkr20Y++6CqX267bsuW1zld5VM2mdaET9VjKKujtEjheNMF9FTBK\nRAqxQX0ucNgSeSIyEtgW6VCdCiQBZbEuLMCIvDSuz9MgEFfypx672ZJKOUSnwd0YExSR24BXsUMh\nHzPGrBeRmyP7FwFfBr4mIgGgHphjemsAvVJKqb45iUkppZwq2klMffYLspVSSrVPg7tSSsUhDe5K\nKRWHNLgrpVQc0uCulFJxSIO7UkrFoV4bCikiJcCuLv56HhDz2a99gBPr7cQ6gzPr7cQ6w9HXe7gx\nptP1W3otuHeHiKyOZpxnvHFivZ1YZ3BmvZ1YZ+i5emtaRiml4pAGd6WUikN9Nbg/3NsF6CVOrLcT\n6wzOrLcT6ww9VO8+mXNXSinVsb7acldKKdWBPhfcRWSmiGwWka0icmdvl6cniMhQEVkhIhtEZL2I\nfCuyPUdEXheRLZH/s3u7rLEmIm4R+VhEXoo8dkKd+4nIUhHZJCIbReR0h9T725H392ci8oyIJMdb\nvUXkMRE5JCKftdjWbh1F5K5IbNssIt36BvA+FdxFxA3cD8wCxgHzRGRc75aqRwSB7xpjxgGnAbdG\n6nkn8IYxZhTwRuRxvPkWsLHFYyfU+ffAK5GvqZyErX9c11tE8oHbgSJjzATsd0XMJf7q/Tgws9W2\nNusY+YzPBcZHfueBSMzrkj4V3IHpwFZjzHZjjB9YDMzu5TLFnDFmvzHmo8jPNdgPez62rv8XOez/\nsN9XGzdEpAC4BHi0xeZ4r3MWcA7wRwBjjN8YU0mc1zvCA6SIiAdIBfYRZ/U2xrwNlLfa3F4dZwOL\njTE+Y8wOYCs25nVJXwvu+cCeFo+LI9viloiMAKYAHwADjTH7I7sOAAN7qVg95R7g+0C4xbZ4r3Mh\nUAL8KZKOelRE0ojzehtj9gK/BXYD+4EqY8xrxHm9I9qrY0zjW18L7o4iIunAs8AdxpjqlvsiX2MY\nN0OdRORS4JAxZk17x8RbnSM8wFTgQWPMFKCOVqmIeKx3JM88G3txGwKkici1LY+Jx3q31pN17GvB\nfS8wtMXjgsi2uCMiCdjA/pQxZllk80ERGRzZPxg41Fvl6wFnApeJyE5suu3fRORJ4rvOYFtnxcaY\nDyKPl2KDfbzX+wJghzGmxBgTAJYBZxD/9Yb26xjT+NbXgvsqYJSIFIpIIrbz4cVeLlPMiYhgc7Ab\njTF3t9j1InBd5OfrgBeOddl6ijHmLmNMgTFmBPbv+k9jzLXEcZ0BjDEHgD0iMjqy6XxgA3Feb2w6\n5jQRSY2838/H9i3Fe72h/Tq+CMwVkSQRKQRGAR92+SzGmD71D7gY+BzYBvyot8vTQ3U8C3urtg5Y\nG/l3MZCL7V3fAvwDyOntsvZQ/WcAL0V+jvs6A5OB1ZG/9/NAtkPq/VNgE/AZ8GcgKd7qDTyD7VMI\nYO/SbuiojsCPIrFtMzCrO+fWGapKKRWH+lpaRimlVBQ0uCulVBzS4K6UUnFIg7tSSsUhDe5KKRWH\nNLgrpVQc0uCulFJxSIO7UkrFof8PnSjdrHAjeWcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x20adbf149e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['training', 'validation'], loc = 'upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2016/2133 [===========================>..] - ETA: 0sTest accuracy:  0.389592124049\n"
     ]
    }
   ],
   "source": [
    "cnn_non_static_best_model = cnn_non_static(embed_dim=embed_dim , num_tokens=num_tokens, max_len=X_train.shape[1])\n",
    "cnn_non_static_best_model.load_weights('best_model.hdf5')\n",
    "cnn_non_static_best_model.compile(optimizer = 'Adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])\n",
    "results = cnn_non_static_best_model.evaluate(X_test, y_test)\n",
    "print('Test accuracy: ', results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CNN-multichannel\n",
    "- A model with two sets of word vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def cnn_multichannel(embed_dim, num_tokens, max_len, windows = [3, 4, 5], filters = 100, p = 0.5, l2 = 3):\n",
    "    inputs = Input(shape = (max_len, ))\n",
    "    embedd_intializer = Embedding(input_dim = len(word_idx) + 1, output_dim = embed_dim, \\\n",
    "                     input_length = max_len, weights = [embedding_matrix], trainable = True)\n",
    "    embedded_1 = embedd_intializer(inputs)\n",
    "    \n",
    "    embedd_intializer = Embedding(input_dim = len(word_idx) + 1, output_dim = embed_dim, \\\n",
    "                     input_length = max_len, weights = [embedding_matrix], trainable = False)\n",
    "    embedded_2 = embedd_intializer(inputs)\n",
    "    \n",
    "    embedded = concatenate([embedded_1, embedded_2])\n",
    "    \n",
    "    conv_results = []\n",
    "    for window in windows:\n",
    "        x = Conv1D(filters, window, activation = 'relu')(embedded)\n",
    "        x = Dropout(p)(x)\n",
    "        x = MaxPooling1D(max_len - window + 1)(x)\n",
    "        conv_results.append(x)\n",
    "    conv_result = concatenate(conv_results)\n",
    "    x = GlobalMaxPooling1D()(conv_result)\n",
    "    x = Dense(50)(x)\n",
    "    outputs = Dense(5, activation = 'softmax')(x)\n",
    "    m = Model(inputs = inputs, outputs = outputs)\n",
    "    m.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['acc'])\n",
    "    return m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cnn_multichannel = cnn_multichannel(embed_dim, num_tokens, X_train.shape[1])\n",
    "callbacks = [ModelCheckpoint(filepath = 'best_model.hdf5', monitor='val_acc', verbose=1, save_best_only = True, mode='max')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "history = cnn_multichannel.fit(X_train, y_train, epochs = 100, callbacks = callbacks, validation_split = 0.1, batch_size = 50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8HNW1wPHf0ao3W81V7rhiDLaFwTgQjIEY00vAEEgg\nCQ4kBFJe8oAUSH0hhUASSuiEBAg49BhMszEdV4wL7k1u6l0raXfP++OurGLJWssry9o9389HH2ln\nZ2fuXe2euXPuvTOiqhhjjIksMd1dAGOMMeFnwd0YYyKQBXdjjIlAFtyNMSYCWXA3xpgIZMHdGGMi\nkAV3Y4yJQBbcjTEmAllwN8aYCBTbXTvOzs7WoUOHdtfujTGmR1q6dGmRquZ0tF63BfehQ4eyZMmS\n7tq9Mcb0SCKyLZT1LC1jjDERyIK7McZEIAvuxhgTgbot596WhoYG8vPz8Xq93V2UiJGYmEhubi5x\ncXHdXRRjzGHUYXAXkUeAc4ACVR3fxvMC3A3MAmqAq1V1WWcKk5+fT1paGkOHDsVt1hwKVaW4uJj8\n/HyGDRvW3cUxxhxGoaRlHgNmHuD5s4CRwZ85wH2dLYzX6yUrK8sCe5iICFlZWXYmZEwU6jC4q+oi\noOQAq5wP/EOdj4DeItK/swWywB5e9n4aE53CkXMfCOxo9jg/uGx3GLYdFRpvddheIA6oElDFIxJV\nwbreF2BTYRXFVfWU1NRT6W1AEGJjBE+MkBAXQ0Ksh8S4GHIzkhmUkUSsp2V7pcEfoKS6nsLKOpLj\nPQzonURinGe/fZXV1PPZznKKqurwxMQQG+P2kxDnISE2hv69EhmSldKpeqgq9f4ANXV+ahr8BAJN\nt7ZMjveQlhhHfGwM9b4Aeyu87KnwUl7TQHW9j9p6P+lJcYzrn87gzGRiYkL7/3sb/Oyt8FLnC3BU\nTup+rwsEdL9lqsqOkloS42LITInf9176A0pVnY/UhFg8Ie7/YNX5/FR5ffgDii+gxMfGkJ2aEPLr\nCyq8ZKTEE+c59DEigYBS2+Cnut5HRa2Poqo6CivrKK2pp64hQJ3PT70vsG/9mBihf69EcjOSyc1I\nYmDvps+hqrKtuIblO0rJSI5n3IB0+qQlHnIZQ3FYO1RFZA4udcPgwYMP565DUlZWxpNPPsm3v/3t\ng3rdrFmzePLJJ+ndu3e76/z85z/n5JNPZvppM2jwB6j3Baiu91NT76e2wY+qEiPS9IVTUJSANgX/\nOE8MWSnx+32IVd0Xwtvgxx9QYj0xxMW4A4EvEMDb4OfdDYV84ajsTh0cdpXV8t7GIj7YWMTiraUk\nxMYwMCOJ3IwkJg/J5IyxfemVfPAdtkVVdSzZWsqSrSXsLKslKzV+3xd6ydZSlm4rpbbBH/L24jzC\noMxkYkSorfdTVeejwttA69sE56QlkJkcT3xsDAmxMeyt9LKjpLbD7Z8yKodrTx7W4fv4wcYiHv9w\nK7vKvBRV1VFUVUeD/8D3Kk6Mi8HbEDjgOinxHkb2TWNETioj+qSQmhDLnnIve8q9FFbVUeH1UVnb\nQElNPWU1Dftel52awPTROUwY1Js1uypYuq2ETYXVDM9OYeLg3ozsk8aqXeV8sKmYwso6AEQgIzme\nBl+AyjofEHx/M5IZnJWMP6DsDu47Od7D6H5pjOyTxrgB6Rw3qDfDs1NQYM2uCt7fVMSeci/9eiXS\nLz0REVi1s5zPdpazqbCaitoG6nz71/3sCf350ZmjGZqdQp3Pz6uf7eGVlbtJT4plaFYK/dIT+Wxn\nOYs2FLKtuIbcjCS+M/0oLp6US0CVt9YW8OKKnZTVNpCZ7L43IlBR20Cl10dZbQOl1fWUVtdT0+xz\n5g+Edl/pxo9A689XnEcYkpXCwN5JrNtTyZ6KlmnR7NR45pwynDmnjAhpP50lodwgW0SGAq+006H6\nd2Chqj4VfLwOOFVVD9hyz8vL09YzVNeuXcvYsWNDLny4bd26lXPOOYdVq1a1WO7z+YiNDe04WO8L\nUFxdt+8frupaj/X+AA2+AP5m77eIkBznITnBgyD7WugAgiDiPkAxIsQIVHp9VNX5EBESY2PQ4PZ9\ngcABP5B7t2/m2pd2M3V4Fr++cDwjclLbXE9Vqa73U1HbwJ4KLwvXFfLGmr2s3V0BuCBx4vBMVCG/\nrJbtxdWU1jQQGyOcdFQ2p43O4YThWYzum9ZmC9MfUJZtL+WttQUs+LyAdXsrAYiPjSE3I4mymgZK\nqusBGNMvjROHZzFpSAZ90xLITIknPSkOVfCr4gseIOt8AarrfGwvqWFTYTVbi6qJiYHk+FiS4z1k\npsSTk5ZAVkoCNfU+8ktryS+toaLWR53Pj7chQGZKPOMH9mJCbi8G9E7CH1D8AaXB77Zf5/OzbFsp\nj32wjaKqOsb2T+eHZ4xixtg+LYL86l3l3PHaOhatL6RPWgLjBqSTnZpAdmoCaYmuPMnxHmKCr1Gg\nNvh+V3gbSE2Io3+vRPr2SiQjOW5fHYqq6li7u4I1uyrYUFDFpsIq9la4IBwbI/RNTyQ7LYFeSXGk\nJcaSkRxHv/RE+vVKQlVZtKGId9YVUOH1kZYQy6QhGYzul8aGvZUs31FGWU0D2akJnDQiixOGZxJQ\nKKyso7iqjvjYGNIT40hNiKW4up7tJdVsK64hzuPOZvqmJ1Lp9bF+byUbCir3HaDSE2MREcpr3UEm\nJd5DdX1TAI2PjWFs/3TG9E2jd0oc6YlxpMR7iIt1Z01bi2t47P2tNPgDnDGuL59sKaG4up6BvZMI\nqDuwgDvzmTo8i7yhmby2ajef5pfTv1ci1XU+Krw++qYnMCQzhdKaekpr6lFl3/uUnhS3r7GUHO++\ng+Ba4qkJHpLjY0lLjCUnNYHstAQykuNJindncrExTWfRPn+APRVe8ktr2V5Sw5aiajYXVrG9pJYR\nOSmcMDyLvCEZlNc2sGZXBWt2V3DKqBzOO3ZAu9/ZAxGRpaqa1+F6YQjuZwM34EbLnAD8RVWndLTN\nIzG4z549mxdffJHRo0cTFxdHYmIiGRkZfP7556xfv54LLriAHTt24PV6uemmm5gzZw7QdCmF4tJy\nZs2axXHHn8inSz+hT7/+/PWRJ0lLTeHWm67njLNmcdFFFzN5/GiuvOoqXpv3XxoaGnj22WcZM2YM\nhYWFXHHFFezatYupU6fyxhtvsHTpUrKzs/eV0dvgp6S6nnpfYF/LwRMjJMZ5SIz14PEIPn8AX8Ad\nKOJiYtiycT0rq1P5w2uf420IcEleLg2+ADvLatlT7qWqzkdNvZ+aeh/NjxEikDckgzPG9eWLo/ow\nqm9qi2CmqnyaX86rq3Yzf9UethbXANA7OY7vnz6Kr500tMX7++O5n/LMknxiY4Tjh2ZyyqgcpgzL\nYPzAXiTEulRJ41lNSsIRNUoXcKmDF1fs4r6Fm9hSVM3Ewb35yglDWLu7gvc3FvH5nkp6JcVxw/Sj\nuGrqkDbTP+FS6W2gtsFPdkpCSKkanz/ArjIvAzOSWqRWVJWiqnqyU+MPOeXnDyibC6tYvr2M5TvK\nCASUk47KYuqILPqkJVJV52NPuZcGf4Cj+qR2mEIpqPBy91sbeH75TqYdlc1Xpw5h2ohsYmIEb4Of\nXWW1DMxI2vfZUVUWri/kkfe2kJkSz8WTcpl2VHaXpZK6S9iCu4g8BZwKZAN7gduAOABVvT84FPJv\nuBE1NcA1qtrhRWM6Cu6/eHk1a3ZVdLSZgzJuQDq3nXt0u883b7kvXLiQs88+m1WrVu0bRlhSUkJG\nRgZFZZVMO+lEHp87j+GD+nHChDG89+HHrN1RwFknTeS9Dz/ixOPzuPTSSznvvPO48sorufrqqznn\nnHO45JJLGDp0KD/84Q/57ne/y7333suyZct46KGHuOGGGxg4cCC33HILr732GmeddRaFhYUtgntn\nNL6vhZV1/Pq/a3hl5W6yUuLJzUiif+8k0hJiSY6PJSXBQ3piHOlJsfRKiuf4oRlkHUTeM7+0ho83\nl/DkJ9tZtbOc9/73NHLS3Ou3FVcz/Y8LuTRvELeePZb0xJ477r7BH2Du0nzufnMDeyq8xMfGkDck\ng5NH5nDFlMGdSlEZE6pQg3uHzSNVvbyD5xX4zkGUrceYMmXKvsCuqvz+T3/mhRdeQFXZvXMnu7Zv\nJqVXbxr8AbYWVyPA0GHDOPF4975PnjyZrVu3trntiy66aN86zz33HADvvfcezz//PAAzZ84kIyMj\nrPXJSUvg7tkT+fOlx4XcMXcwcjOSyZ2czMTBvTn9znd46N3N3DLLHawfWLSZ2JgYfnDGqB4d2MH1\nfVw+ZTAXThzIuj2VjO6X1qWtdGM648g79w06UAv7cElJSUFVKa9t4OXX3mT+G2/w9MtvMrhvby46\n+0tkJcUwqm8aMSLEe2LIyUgmMaGppevxeKitbbujLiG4nsfjwefzHZb6NOqKwN7c8JxUzj12AE98\ntI1vfXEEvkCAZ5fmc/HkXPqkH56RAodDYpyHYwe134luTHeya8s0k5aWRmVlZYtlu8pcJ0llRQV9\ns7M4dlgfCnZs4aOPPgLcF9wTIwzPSSXhEFtv06ZN45lnngHg9ddfp7S09JC2151umH4UtQ1+Hn5v\nM4++vxWfP8C3Thne3cUyJmocsS337pCVlcW0adMYP348SUlJ9OnTh5KaBjKS4/n67At4+d+PM27c\nOEaPHs2JJ54Y9v3fdtttXH755TzxxBNMnTqVfv36kZaWFvb9HA4j+6Yxa3x/Hv9gGwLMOqY/Q7M7\nN07cGHPwQhot0xWOxNEyrRVWetld7mVU38OTU62rq8Pj8RAbG8uHH37I9ddfz4oVKw55u931vq7d\nXcFZd78LwCvf/QLjB/Y67GUwJtKErUM1WqkqJdX1pMTHHrbOsu3bt3PppZcSCASIj4/nwQcfPCz7\n7Spj+6dzWd4gvD6/BXZjDjML7u2oqvNR5wvQN/PwdQCOHDmS5cuXH7b9HQ53XDKhu4tgTFSyDtV2\nlFTXExsTQ3pSzx62Z4yJThbc29DgD1BR6yMjJW7fVHFjjOlJLLi3oaS6HkXJTI7v7qIYY0ynWHBv\nJRDsSE1NiD3kcevGGNNdLLi3UlZTT4M/sO+aKAeSmuqurrhr1y4uueSSNtc59dRTaT3ks7W77rqL\nmpqafY9nzZpFWVnZQZTaGGNasuDejKpSWFlPUpyH1IO4KuGAAQOYO3dup/fbOrjPmzfvgNeGN8aY\njlhwb+YH//NjHn/ofnLSEhARbr/9dn79618zY8YMJk2axDHHHMOLL7643+u2bt3K+PHuasi1tbXM\nnj2bsWPHcuGFF7a4tsz1119PXl4eRx99NLfddhsAf/nLX9i1axfTp09n+vTpgLuEcFFREQB33nkn\n48ePZ/z48dx111379jd27FiuvfZajj76aM4888x2r2FjjIlOR+4491dvhj2fhXeb/Y6Bs37X5lOq\nyqmzzufXP72Z3/zkfwB45plnmD9/PjfeeCPp6ekUFRVx4oknct5557V77ev77ruP5ORk1q5dy8qV\nK5k0adK+537zm9+QmZmJ3+9nxowZrFy5khtvvJE777yTBQsW7Hdp36VLl/Loo4/y8ccfo6qccMIJ\nfPGLXyQjI4MNGzbw1FNP8eCDD3LppZfyn//8hyuvvDJMb5QxpqezlntQdZ2PYaPHU15SxO7du/n0\n00/JyMigX79+3HrrrUyYMIHTTz+dnTt3snfv3na3s2jRon1BdsKECUyY0DSJ55lnnmHSpElMnDiR\n1atXs2bNmgOW6b333uPCCy8kJSWF1NRULrroIt59103nHzZsGMcddxxw4EsLG2Oi05Hbcm+nhd1V\niqrcpKXLLv0yc+fOZc+ePVx22WX861//orCwkKVLlxIXF8fQoUPxer0db7CVLVu28Mc//pHFixeT\nkZHB1Vdf3antNEoI8dLCxpjoZC13XEqmqs5H7+Q4Zs+ezdNPP83cuXP58pe/THl5OX369CEuLo4F\nCxawbdu2A27rlFNO4cknnwRg1apVrFy5EoCKigpSUlLo1asXe/fu5dVXX933mrYuNQxw8skn88IL\nL1BTU0N1dTXPP/88J598chhrboyJVEduy/0w8jb4CaiSHO9hwNFHU1lZycCBA+nfvz9f+cpXOPfc\ncznmmGPIy8tjzJgxB9zW9ddfzzXXXMPYsWMZO3YskydPBuDYY49l4sSJjBkzhkGDBjFt2rR9r5kz\nZw4zZ85kwIABLFiwYN/ySZMmcfXVVzNlirsl7Te/+U0mTpxoKRhjTIfskr9AcVUdO8tqGdMvjfjY\nyJu4dKRdStkY03mhXvI3pLSMiMwUkXUislFEbm7j+QwReV5EVorIJyIyvjOF7i419X5iY2I6vBu7\nMcb0FB1GMxHxAPcAZwHjgMtFZFyr1W4FVqjqBOCrwN3hLmhXqqn3kxzvaXd4ozHG9DShNFWnABtV\ndbOq1gNPA+e3Wmcc8DaAqn4ODBWRvp0p0OFOE/n8Aep8fpITIi8dA4f//TTGHBlCCe4DgR3NHucH\nlzX3KXARgIhMAYYAua03JCJzRGSJiCwpLCzcb0eJiYkUFxcf1oBUU+8HIDk+8vqWVZXi4mISEw/f\nDUeMMUeGcEW03wF3i8gK4DNgOeBvvZKqPgA8AK5DtfXzubm55Ofn01bg7yoVtQ1Uen14KhIj8trt\niYmJ5Obud5w1xkS4UIL7TmBQs8e5wWX7qGoFcA2AuMT1FmDzwRYmLi6OYcOGHezLDslVD39McVU9\n826a1PHKxhjTQ4SSllkMjBSRYSISD8wGXmq+goj0Dj4H8E1gUTDgH9ECAWXF9jImDbErMBpjIkuH\nLXdV9YnIDcB8wAM8oqqrReS64PP3A2OBx0VEgdXAN7qwzGGzsbCKyjofEwdldHdRjDEmrELKuavq\nPGBeq2X3N/v7Q2BUeIvW9ZZvLwVg4mBruRtjIktUz9pZtq2M3slxDMtO6e6iGGNMWEV1cF+xo4zj\nBvW2yUvGmIgTtcFdVdleUsNROandXRRjjAm7qA3ulXU+ahv89E23CT7GmMgTtcG9oMLdKKNPekIH\naxpjTM8TxcG9DoA+adZyN8ZEnqgN7nsrXcu9r7XcjTERKHqDe2PL3XLuxpgIFMXB3UtqQiypCZF3\nNUhjjIna4F5QWUefNEvJGGMiU/QG9wqvjZQxxkSsqA3ueyvqbIy7MSZiRWVwV1X2VngtuBtjIlZU\nBvcKr486X8By7saYiBWVwb1pdqq13I0xkSkqg3vjGPe+1nI3xkSoKA3ujbNTreVujIlMURncCyob\nZ6day90YE5lCCu4iMlNE1onIRhG5uY3ne4nIyyLyqYisFpFrwl/U8Nlb4SUtIZbkeJudaoyJTB0G\ndxHxAPcAZwHjgMtFZFyr1b4DrFHVY4FTgT+JSHyYyxo2BZU2gckYE9lCablPATaq6mZVrQeeBs5v\ntY4CaeLuV5cKlAC+sJY0jGwCkzEm0oUS3AcCO5o9zg8ua+5vwFhgF/AZcJOqBsJSwi5QUOm1Me7G\nmIgWrg7VLwErgAHAccDfRCS99UoiMkdElojIksLCwjDt+uC42anWcjfGRLZQgvtOYFCzx7nBZc1d\nAzynzkZgCzCm9YZU9QFVzVPVvJycnM6W+ZCU1zZQ7wvYBCZjTEQLJbgvBkaKyLBgJ+ls4KVW62wH\nZgCISF9gNLA5nAUNl30TmKxD1RgTwTocC6iqPhG5AZgPeIBHVHW1iFwXfP5+4FfAYyLyGSDA/6pq\nUReWu9MKgrfXs3unGmMiWUgDvVV1HjCv1bL7m/29CzgzvEXrGtZyN8ZEg6ibodp46QFruRtjIlnU\nBfeCCi/pibEkxXu6uyjGGNNloi+4V9bZSBljTMSLuuDu7sBk+XZjTGSLwuBeR1/LtxtjIlxUBXdV\npbCyjhxruRtjIlxUBfeymgbq/QEbKWOMiXhRFdz33aTDLhpmjIlwURbcG8e4W3A3xkS2qAruhftu\nr2dpGWNMZIuq4G5pGWNMtIiu4F5RR0q8h5QEu3eqMSayRVdwr/RaSsYYExWiLLjXkWMpGWNMFIiq\n4F5YWWf5dmNMVIiq4F5Q4bUJTMaYqBA1wb26zkd1vd/SMsaYqBA1wd2GQRpjoklIwV1EZorIOhHZ\nKCI3t/H8j0RkRfBnlYj4RSQz/MXtvILGOzDZRcOMMVGgw+AuIh7gHuAsYBxwuYiMa76Oqv5BVY9T\n1eOAW4B3VLWkKwrcWU0td8u5G2MiXygt9ynARlXdrKr1wNPA+QdY/3LgqXAULpwsLWOMiSahBPeB\nwI5mj/ODy/YjIsnATOA/h1608Cqo9BLviaF3clx3F8UYY7pcuDtUzwXeby8lIyJzRGSJiCwpLCwM\n864PrLDCTWASkcO6X2OM6Q6hBPedwKBmj3ODy9oymwOkZFT1AVXNU9W8nJyc0EsZBjY71RgTTUIJ\n7ouBkSIyTETicQH8pdYriUgv4IvAi+EtYnjY7FRjTDTpMLirqg+4AZgPrAWeUdXVInKdiFzXbNUL\ngddVtbprinpo3EXDLLgbY6JDSNe+VdV5wLxWy+5v9fgx4LFwFSyc6n0BSmsabBikMSZqRMUM1cIq\nGwZpjIkuURHcbXaqMSbaREdwt9mpxpgoE2XB3VruxpjoEBXBvbDCiwhkpsR3d1GMMeawiIrgXlBZ\nR1ZKArGeqKiuMcZET3C3lIwxJppESXC3CUzGmOgSHcG9wlruxpjoEvHB3R9QiqvrbRikMSaqRHxw\nL66uwx9QS8sYY6JKxAf3ggqbwGSMiT6RH9wr3aUH+lrL3RgTRSI/uDe23NOt5W6MiR4RH9z3BoN7\nTqq13I0x0SPig3tBpZfMlHjiYyO+qsYYs0/ER7y9NsbdGBOFIj64F1Z6Ld9ujIk6ER/c91bU0dda\n7saYKBNScBeRmSKyTkQ2isjN7axzqoisEJHVIvJOeIvZOYGAUlhVZxOYjDFRp8MbZIuIB7gHOAPI\nBxaLyEuquqbZOr2Be4GZqrpdRPp0VYEPRnF1Pf6A0tfSMsaYKBNKy30KsFFVN6tqPfA0cH6rda4A\nnlPV7QCqWhDeYnZO4wQm61A1xkSbUIL7QGBHs8f5wWXNjQIyRGShiCwVka+2tSERmSMiS0RkSWFh\nYedKfBBsApMxJlqFq0M1FpgMnA18CfiZiIxqvZKqPqCqeaqal5OTE6Zdt89a7saYaNVhzh3YCQxq\n9jg3uKy5fKBYVauBahFZBBwLrA9LKTtp3+xUC+7GmCgTSst9MTBSRIaJSDwwG3ip1TovAl8QkVgR\nSQZOANaGt6gHr6DSS0ZyHAmxnu4uijHGHFYdttxV1SciNwDzAQ/wiKquFpHrgs/fr6prReQ1YCUQ\nAB5S1VVdWfBQ7K2os5EyxpioFEpaBlWdB8xrtez+Vo//APwhfEU7dAWVdZaSMcZEpYieoVpQ4bWW\nuzEmKkVscA8ElMJKu2iYMSY6RWxwL6mpx2ezU40xUSpig3vTvVOt5W6MiT4RG9z3Nk5gspa7MSYK\nRWxwL7SWuzEmikVscN9b0dhyt+BujIk+kRvcK730ttmpxpgoFbHBvaCijr5plm83xkSniA3ueyvt\nDkzGmOgVscG9sMJLH2u5G2OiVEQG90BAKbCWuzEmikVkcC8Nzk61YZDGmGgVkcF9d7kbBtm/l6Vl\njDHRKSKD+55gcO/XK6mbS2KMMd0jIoP77vJaAAZYy90YE6UiNLh7iY0RslIt526MiU4RGdz3lLub\ndHhipLuLYowx3SKk4C4iM0VknYhsFJGb23j+VBEpF5EVwZ+fh7+oodtVXmudqcaYqNbhPVRFxAPc\nA5wB5AOLReQlVV3TatV3VfWcLijjQdtT7mX8wF7dXQxjjOk2obTcpwAbVXWzqtYDTwPnd22xOk9V\n2V3uZUBvGyljjIleoQT3gcCOZo/zg8taO0lEVorIqyJydFhK1wmlNQ3U+QL0s5t0GGOiWIdpmRAt\nAwarapWIzAJeAEa2XklE5gBzAAYPHhymXbe0qyw4DLK3BXdjTPQKpeW+ExjU7HFucNk+qlqhqlXB\nv+cBcSKS3XpDqvqAquapal5OTs4hFLt9NoHJGGNCC+6LgZEiMkxE4oHZwEvNVxCRfiIiwb+nBLdb\nHO7ChmJ38A5MNoHJGBPNOkzLqKpPRG4A5gMe4BFVXS0i1wWfvx+4BLheRHxALTBbVbULy92u3WW1\nNoHJGBP1Qsq5B1Mt81otu7/Z338D/hbeonWOTWAyxpgInKFqE5iMMSYCg/ueci/9LLgbY6JcRAV3\nm8BkjDFORAV3m8BkjDFORAX3fddxtwlMxpgoF1nBvcwmMBljDERacK+we6caYwxEWnAPTmDKtglM\nxpgoF1HB3SYwGWOME1HB3SYwGWOME1HB3SYwGWOMEzHB3SYwGWNMk4gJ7jaByRhjmkRMcG+8A5Pl\n3I0xJoKC+9biagCGZKV0c0mMMab7RUxw31LogvvQ7ORuLokxxnS/yAnuxdX0S08kOT5c9/w2xpie\nK2KC+9aiamu1G2NMUEjBXURmisg6EdkoIjcfYL3jRcQnIpeEr4ih2Vpcw7Bsy7cbYwyEENxFxAPc\nA5wFjAMuF5Fx7ax3B/B6uAvZkfLaBkqq6xlqnanGGAOE1nKfAmxU1c2qWg88DZzfxnrfBf4DFISx\nfCHZWtTYmdpGcPeWw6Nnw4Y3DnOpjDGm+4QS3AcCO5o9zg8u20dEBgIXAveFr2ihaxwG2WZa5vWf\nwbb3YOUzh7lUxhjTfcLVoXoX8L+qGjjQSiIyR0SWiMiSwsLCMO0aNhdWIwKDM1t1qG5eCMseB08C\nbP8wbPszxpgjXSjBfScwqNnj3OCy5vKAp0VkK3AJcK+IXNB6Q6r6gKrmqWpeTk5OJ4u8v63F1Qzo\nlURinKdpYX01vHQjZI6A034C5TugbHvY9mmMMUeyUAaFLwZGisgwXFCfDVzRfAVVHdb4t4g8Bryi\nqi+EsZwHtLWoev+UzFu/grJtcM2rEJ/qlm37EHoPPlzFMsaYbtNhy11VfcANwHxgLfCMqq4WketE\n5LquLmBHVJUtrce4b1kEH98Px18LQ06CvkdDQi/Y9n73FdQYYw6jkKZzquo8YF6rZfe3s+7Vh16s\n0JXWNFBTTAhHAAAW3UlEQVTh9TUNg6wpgee+BVkj4PTb3bIYDww+wfLuxpio0eNnqG4pajZSRhVe\n+i5UF8LFD0NCatOKQ06CovVQFb6OXGOMOVL1+ODeYoz70sfg81fg9NtgwHEtVxx8kvttrXdjTBTo\n8cF9S1E1MQKDYivgtVtg+HQ48Tv7rzhgIsQmWnA3xkSFnh/ci6sZlJlM/LaF4KuFM38FMW1UKzYe\nco+3TlVjTFTo8cF9a1G160zd+h4kZUKfo9tfefBU2PMZeCsOXwG7i7cC9q7u7lIYY7pJjw7uqto0\nxn3re67TtK1We6MhJ4EGIP+Tw1fI7qAK/74S7j8Z8pcc+raWPArL/uHe4/KdUFsa/CkLT3mNMWHX\no+9sUVhVR3W9n6OTy92EpRO/feAX5B4PMbFuHPxRp7e/3pqXoKEWjr0stIL4fbB3FQT87nFcIuSM\ncUMwAQIB2LwANr0NR18IuXmhbbez1rwAW95xfQz/+QZ8611ITO/ctta/Bq98r/3nT78dvvD9zm3b\nGNNlenRw31pUA8B43yq3YOi0A78gIRWGnwqrnocZt7fdyleF+bdCxU7IGAKDT+y4IEsegVd/1HJZ\nUobr3M0eBZ89CyWb3PIP/wZHXwQzfgY1pbDpLdj+EQycDHnXQPqAjvd3IHVV8Nqt0O8YOOv38NjZ\nMO9HcNHfwd/gRhRtehvGnuvKEXeAG4qrwqI/ulm9V73gDqClW8FX557/5AHY8KYFd2OOQD06uG8u\nrAJgUMUySOx94Hx7owmXwXPXwo6PXJqmteKN7jo04oH/XAvXvQtJvd1zO5e5dMRRM1q+ZusiSM+F\nc/7sHteWuIuWbXobVj8Hg06AU292Zwsf3w/v/8UtB0Ag6yi37rt/gjFnQ59ml8tPyYbMYZAxzB0w\nGsWnuk7i1hb9ASp3wZcfcxO3vngzLPyt287611z9krNh3TyY/xOY+BUYfbY7q/G0+jhseQd2LoGz\n73STwrJGtHy+dJs7WPgbwBPXtHzej9yZz9l3tl1GE70+eRDWvgxfeRZiE7p+f5vedt+VQVO6fl9H\nmJ4d3IuqiY+NIWX3RzBk2oHz7Y3GnA1xKbDy320H941vud8XPQDPfwte+b77+507XPCNiYObtze1\neFVhxyfujGDUmU3bOXa2e66m2AXWRtNvhclXw/J/uoA9Yrp7vmQLLH3ULV/7Usf1kBjoleu2kTnc\nHQCSs+DDe+C4r7jADnDyD11K6MO/QfZouOIZGHmmS00tfhA+vBc++Ku7PMOI6XDGL90ZC7j6pvZz\n22vLoOPh4/tcSmrARLesvjoY8OuhugguffzwfImjQXUxiEBy5sG/tqEWKne7z0p3yV8Cr/4vqN+d\n9Z303a7dX20pPHW5O9Oc+h047WcHPlONMD07uBdWkZdRg5RugSlzQntRfAqMPQdWP+/SFq0Dz6a3\n3BfgmEtcGuKtX7rWa9l2yJ3iOmPzP4Fhp7j1y7ZD1V7X8m1NpGVgb5Q+AL7445bLMoe5wHrGL5uW\nqbptl2x2wb++qml5bUlw+WZY86J7DC5In/6Lpm14YuGyf7ohoKPPbmqdD/+i+6kthc3vuHqvfgG2\nvguX/aupb+LM37T/hRgUPIDs+KQpuG//0AX28ZfAqrnw9BVu/3FJbW8jEvkb3Hs67OTQDmyqwbOk\nZe7/WboVUvu6M8QRp7nHix9y/5/UvjBnIaQ2u6pq5V73WW3dOq0thRVPwsY3YdsH4PPCiBlw5q+h\n7zg3ourTp93Bf9JXYfRZ4XsPWvNWuP6f9IGu8bDoD67R0JkDVahWPuvqPPZc17jZ9LZrqPU7puv2\neQTp4cG9miuTN0IlMPQLob9wwqWu5b7hdfePb+SrcyNCGluq077nvqR7PoNLn3Ct8zuGuqDXGNx3\nBEfeNAa6cBKBtH7up62zjOZqy6B0CyT2avnFB3eAGdfWzbNwqZ6jL3A/J90IT14Gj5/rDjZJma4f\noD29ciFtgHsPTviWW7Z5IXji4by/uoPHSzfCs1fD7KdanlntXe3e62GnuM5nkbb34fftny5qrmyH\nO80/+oJD76/YvBDqa2DMrM5vo6bE1XfLO9B3vAsmfQ+QLty13N1QZuu77nFKHxf8tixyB8dGCelw\n3BXuc/vs1fDVF1wqrHgTPH4eVOTDxKtg5v9BQhpsWgAvfNul6LJHQ97X3f/zw7/C/dNcf9COj12D\nIbGXS9NN+hp86bctL9sRLvN+5BpC17zq6nL/NBfgZ/5f+PcF7oC59DHX6Ljsn7D+dXjxO/DgaXDa\nT2HqDc0GPPjd+gf6nPVAPbY2Df4A20tqmJy82n04D/QFam3Yqe5LtPLfLYP79o+goaYppx7jgSv/\n4/75ja3XARPdF6/Rjo9dmqfPfreVPbySekPSxEPbRvZI+OabTcFp+k/cmc6BDJrSdIAD2LTQHeji\nk11rsKEWXv2x62uYGhzNVLEbnrjQnZWAO0CMv8idtcQ0uyb/0sdg/k9daqd1P8emBe7Ufv1rbnjr\n+3fD5U/BwEkHX+/6anj9p65jHNxB7vTbW5YlFEUb3MGxfIfrZF7+T3jgVJjxczdruvnBra7KvS8r\n/uXSabP+6FJ5CWnu+UDApbs2L3D9SeMvdkF3yEkuXfjGz2HyNfCP81yj5PhvuvI3NjyWP+E682e/\n7TrrGx3/DddJ/tmzMPY8mPJNdxBa8Fv3Hm5ZBFOudf1D2aPaP+g2qtzr/teN5W7N73MHlJVPw6m3\nNA1QmHily79PubZrUkX5i6FgNZx7t3s86kz49kfwyk3uvVv3Gkz4smu8bV4Aaf3dqLL2+oh89a6f\nrniTa0RV7HKfO3ANpOk/aXkW0lAL7/3Z1bObLjPeY4P7jpIafAFlePWKYL79IL6InliXdln8kDt1\nbeyo3PSWy6kPPbnZunEtOwuHnQIf/MV9ORNSXYomd3LkHPWTM90BbcPrcNQZHa8/aIobelm5x3VC\n7/3M5TYbTZnjWsRv3ubOrnJGwzNfde/fVS+41tz6+e602RPXdCXPgrUuP+uvh39fBVe/4gK3v8Et\nX/Kw6xj+wvfddl+6CR49Cy64zx0oOlJf476khevg7V+5tNdJ33XLP/iL63i+6IH2g1ZrxZvgoRnu\n8/O1l10Qm3oDvHyTO3Csn+/K1nuQO9t46nIXfKbd5PpFEnu13F5MDPSf4H6aO3Y27FoBH90Ly//l\ngtHV/3VplvGXuMC//Ak44Tr3XrZOhyVnwszfup/mzvgFjPoS/Pd/3Gix+bdCr0HuoHHCdfun5mpL\n3UHikwcgayR84/WWLX5V9xl6/WdQtA5GfglO/p+m56f/BD6bC2/eDpf+o+W2dy5zrfrTb3efl85Y\n+pjrSB1/cdOylCx3Bv7pUzDvx7D9AxfUh3wB1v3XHRxPbHUVc1V3vao3fu5SZuDOTNMHuNQluM+O\ntxwufqjpdQt/B+/f5c5QZ/+rc3U4VKraLT+TJ0/WQ/HG6j162s1/V70tXfX9vx78BnYud6/95MGm\nZfdOU3307AO/buPb7nXr31Ctq1K9PUP1zV8e/P4jxY7F7v1Y/aLqymfd3zuWtFynqkj1D6NU/5qn\n+vy33Tqrnm+5zks3Bpc/p1pfq3rPVNXfj1Dd9anqneNV7xiumr9E9bFz3Xrzf6ra4G16fWWB6kNn\nuudevEG1Yk/b5d34tur9p7j1Gn/+NE5186KmdT5+wP1f/3yM6gf3qNaUdPw+vP8Xt63C9S2XBwKq\nS/+h+psBqr/NVX3nD6q/P8r9vf6NjrfbFl+96uPnqf5xtGrB5y2f81aq7l3bue02Kt2muvgR1cfP\nd3W682jVT/+tunOZ6mdzVd/6tervhqje1kv1qStUb++t+szXXF0by/D0V9xr756ouublpueaW/A7\nt86C/2t6vnCD6h3D3PLfj1Dds7rla+pr99+O36+69HH3+VBVrSlV/VVf95lqT1Whe+8CAffz2Lmq\nvxvqXtuocL3qwzNdWf42xX1mS7er+n0tt/X2b906n7/qHu9c5j4/fxjplu9c1n45OgFYoiHEWHHr\nHn55eXm6ZEknZ096K1j+1G2M3foECXGxyPXv7z9MryOq8OB013K7/CnIGQt/GgUzboOTf9D+6+pr\n4I4hrjUz8kx4/By44tmWI2Wiia8e/i/XnV57y91Inx9v2f9MavNC+McFgLrW9um3t9pOHTx2jktF\nHDXD5dEb39eiDfDwma7T2BPvTrWPu4L9+OpcB/jHf3frTbupqW+koQY+ug82vuFOkydeFRxlNNyl\n1Fq3TLcscnfzyv8EYpNciumMX7bfuTz367D9Y/hBO5d8KN0Kz1/nOpwzhrpRS51tlYJLFfobun70\nx+Z33JnHnpUtlw+f7q7j1O8YeO8ud2Z2+i/cWdNTl0PBGvddmvqdlme+revw4g3w6ZPwhR+4fpuH\nz3BpsvPvdZPnfHVw1fNu5NXiB93ZwAnXuU7hGI9LX71yk5tBDXDMl90Zx3t3uo7nASGmKnevhL+f\nAtNudP/nPavgH+cD6s5EJ17V/tm5rx4eCA5OuO4997qaYvjGG/D3k2FgHlw5t2ndF7/thmSPDOHM\nuA0islRVO5wJ2fOC+8Y33ZekupB5cjKzbrq38zmtqgLXGVW6xZ3SrvgnfGsR9D/2wK979GzXETX2\nXHdK/+MtXdvrf6R7+Ex3sKzc7S61fNk/217vw3vcwfScP7edRqvY7b4kVXvhhOvhrN81PZe/BN76\nBZx6KwyZeuDyFG9yp/uth5Qm9oJTfuRSRaEOz9z9qUs9LP+n+5LOfhLS+u6/3t3HQb/x7dcdXDBb\n96rLm/ekz0sgABvmQ8AXHHo7rGVfjCrMvcaN2krs7er55UcOPAu8+bZf+Z67kX1SphvdcvUrrp+g\nZLP7fpbvcOum9HGpufWvuZThxQ+6CXufPukGP8TEuvSezwv9j4NvvXNw9Xz+Olj1HFzysLsvRFyy\nS7GF0nDcuRQeOt31H1XkuwEEY2a5vPubt8PXX3ffjWe+ButfhZl37J8CClHkBveiDfDfH3BLxcVs\niB3F3Os7GEXSkeoi16Lc+5nL4f7Pho7Hyy/8nfsZdIJrTd6w+NDK0NO9/lP44G+Awtl/cnnaztq1\nwnX2hWNM8t41TZ224A7anQ2qa1+G5+a4AHT5ky0bALWlbhTVjJ+7/Hk0qq+GR77kfl/+9MGdlQQC\n8NrNbp7H7KdgZLODQtl2WHiHm4Mx9jzXx7DkETf6JjbRNbKm/6RpaHF5vmtEjJ7lhqIejPJ8+Otk\nd3DoNRi+9pI7kIXq9Z+5/prxF8Mlwc75+mq4+1g3Yiku0TVOZ/3Rnel2UuQG98bX//oNThvTh99f\n0kErOxQ1Je60euBkd1mAjmz7wHXeARx3JVxwz6GXoSdb8xI8c5X7+7vLDj5F1lPs/tSlHOqqXPql\nsbN109tu9M9VL7ggFK189W5yXWcHFzTUhj4fYssil9I5/psulRIu7//FNS5m/+vgMwINXteZfcwl\nLWeTf3gvzL8FEJdSnPy1QypiqME9pP+CiMwE7gY8wEOq+rtWz58P/AoIAD7ge6r63kGXOkTltQ0U\nVdUzPCdM43GTM9244VANzHN5WF9tVE5r3k/je9BrUPfOgOxq/Y+FC//u+lk2LYBx57nlO5e5363v\n/hVtDvVSEwcz0W3YKfC9lR2vd7Cm3dj5g0VcYtst8ryvu76Wsee54ZeHSYfz9UXEA9wDnAWMAy4X\nkdaDut8CjlXV44CvAw/RhRrvmzo8u4Mx2F0lNr4p72vB3U2y6neM64PoaFx0Tzd4qsvdr3+tadmu\n5e6g1ry1ZkyjuES47InDGtghtJb7FGCjqm4GEJGngfOBNY0rqGpVs/VTgC7N9TReMCxsLffOmDAb\n6ipdLs3AtQvcKXmk88S6zrz1812uOCbGBffBHXTyGnOYhfJtHAjsaPY4P7isBRG5UEQ+B/6La713\nmc2F1XhihMGZyV25mwM79jI3mzOUi5VFA0/cwc/o7KlGnwU1RW6EROVed3nozsyMNaYLhS0yqerz\nqjoGuACXf9+PiMwRkSUisqSwsLDT+9pcVMWgjCTiYy2wmm5w1Aw3G3f9q67VDqGPpzbmMAklOu4E\nBjV7nBtc1iZVXQQMF5H9Loeoqg+oap6q5uXk5LTx6tBsLqzu3pSMiW5JGS4Ns34+7Frm0lH9JnT8\nOmMOo1CC+2JgpIgME5F4YDbQYnaIiBwl4nrSRGQSkAAUh7uwAIGAsqWouvs6U40BGD3TzaZd+4rr\nd+mKKykacwg6DO6q6gNuAOYDa4FnVHW1iFwnIo1TrC4GVonICtzImsu0iwbQ7yyrpc4XsJa76V6j\nZrrfBast326OSCGNc1fVecC8Vsvub/b3HcAd4S1a2zY3DoPMsZa76UbZIyFzhLs3ruXbzRGox/VI\nxntimHZUFiOs5W66W+OdiwZYy90ceXrcRcinjshi6ois7i6GMe4CZDGeji80Z0w36HHB3ZgjRsaQ\nlve8NeYI0uPSMsYYYzpmwd0YYyKQBXdjjIlAFtyNMSYCWXA3xpgIZMHdGGMikAV3Y4yJQBbcjTEm\nAnXbDbJFpBDY1smXZwNFYSxOTxGN9Y7GOkN01jsa6wwHX+8hqtrhNdO7LbgfChFZEsrdvyNNNNY7\nGusM0VnvaKwzdF29LS1jjDERyIK7McZEoJ4a3B/o7gJ0k2isdzTWGaKz3tFYZ+iievfInLsxxpgD\n66ktd2OMMQfQ44K7iMwUkXUislFEbu7u8nQFERkkIgtEZI2IrBaRm4LLM0XkDRHZEPyd0d1lDTcR\n8YjIchF5Jfg4GurcW0TmisjnIrJWRKZGSb2/H/x8rxKRp0QkMdLqLSKPiEiBiKxqtqzdOorILcHY\ntk5EvnQo++5RwV1EPLgbcJ8FjAMuF5Fx3VuqLuEDfqiq44ATge8E63kz8JaqjgTeCj6ONDfhbsTe\nKBrqfDfwmqqOAY7F1T+i6y0iA4EbgTxVHQ94gNlEXr0fA2a2WtZmHYPf8dnA0cHX3BuMeZ3So4I7\nMAXYqKqbVbUeeBo4v5vLFHaqultVlwX/rsR92Qfi6vp4cLXHgQu6p4RdQ0RygbOBh5otjvQ69wJO\nAR4GUNV6VS0jwusdFAskiUgskAzsIsLqraqLgJJWi9ur4/nA06pap6pbgI24mNcpPS24DwR2NHuc\nH1wWsURkKDAR+Bjoq6q7g0/tAfp2U7G6yl3Aj4FAs2WRXudhQCHwaDAd9ZCIpBDh9VbVncAfge3A\nbqBcVV8nwusd1F4dwxrfelpwjyoikgr8B/ieqlY0f07dMKeIGeokIucABaq6tL11Iq3OQbHAJOA+\nVZ0IVNMqFRGJ9Q7mmc/HHdwGACkicmXzdSKx3q11ZR17WnDfCQxq9jg3uCziiEgcLrD/S1WfCy7e\nKyL9g8/3Bwq6q3xdYBpwnohsxaXbThORfxLZdQbXOstX1Y+Dj+fign2k1/t0YIuqFqpqA/AccBKR\nX29ov45hjW89LbgvBkaKyDARicd1PrzUzWUKOxERXA52rare2eypl4CvBf/+GvDi4S5bV1HVW1Q1\nV1WH4v6vb6vqlURwnQFUdQ+wQ0RGBxfNANYQ4fXGpWNOFJHk4Od9Bq5vKdLrDe3X8SVgtogkiMgw\nYCTwSaf3oqo96geYBawHNgE/6e7ydFEdv4A7VVsJrAj+zAKycL3rG4A3gczuLmsX1f9U4JXg3xFf\nZ+A4YEnw//0CkBEl9f4F8DmwCngCSIi0egNP4foUGnBnad84UB2BnwRj2zrgrEPZt81QNcaYCNTT\n0jLGGGNCYMHdGGMikAV3Y4yJQBbcjTEmAllwN8aYCGTB3RhjIpAFd2OMiUAW3I0xJgL9P555+xmA\nBuEWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x20adc065390>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['training', 'validation'], loc = 'upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2133/2133 [==============================] - 0s     \n",
      "Test accuracy:  0.407407407687\n"
     ]
    }
   ],
   "source": [
    "cnn_multichannel_best_model = cnn_multichannel(embed_dim=embed_dim , num_tokens=num_tokens, max_len=X_train.shape[1])\n",
    "cnn_multichannel_best_model.load_weights('best_model.hdf5')\n",
    "cnn_multichannel_best_model.compile(optimizer = 'Adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])\n",
    "results = cnn_multichannel_best_model.evaluate(X_test, y_test)\n",
    "print('Test accuracy: ', results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results\n",
    "- All four model variations show high level of training accuracy (almost 1.0), but suffer from low level of validation/test accuracy (below 0.4)\n",
    "- Test accuracies\n",
    "    - ```CNN-rand```: 0.390060947037\n",
    "    - ```CNN-static```: 0.341303328715\n",
    "    - ```CNN-non-static```: 0.389592124049\n",
    "    - ```CNN-multichannel```: 0.407407407687\n",
    "- So with this dataset, ```CNN-multichannel``` model performs best"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
